Top Banner
PBS Works is a brand of Altair 2019.1
336

Control Administrator's Guide - Altair PBS Works · Altair Control 2019.1 Administrator's Guide iv • Altair Embed SE™ ©1989-2019 (formerly solidThinking Embed® SE) • Altair

Jun 13, 2020

Download

Documents

dariahiddleston
Welcome message from author
This document is posted to help you gain knowledge. Please leave a comment to let me know what you think about it! Share it to your friends and learn new things together.
Transcript
Page 1: Control Administrator's Guide - Altair PBS Works · Altair Control 2019.1 Administrator's Guide iv • Altair Embed SE™ ©1989-2019 (formerly solidThinking Embed® SE) • Altair

PBS Works is a brand of

Altair 2019.1

Page 2: Control Administrator's Guide - Altair PBS Works · Altair Control 2019.1 Administrator's Guide iv • Altair Embed SE™ ©1989-2019 (formerly solidThinking Embed® SE) • Altair

Altair Control 2019.1 Administrator's Guide

Intellectual Property Rights Notice: Copyrights, Trademarks, Trade Secrets, Patents and Third Party Software Licenses

Updated: April 30, 2019.

Altair® PBS Works® v.2019.1Accelerating Innovation in the Cloud™ Copyright© 1994-2019 Altair Engineering Inc. All Rights Reserved.

PBS Works - Accelerating Innovation in the Cloud™Altair PBS Professional™ ©1994-2019Altair Control™ © 2008-2019; (formerly PBS Control)Altair Access™ © 2008- 2019; (formerly PBS Access)Altair Accelerator™ © 1995- 2019; (formerly NetworkComputer)Altair Accelerator Plus™ © 1995- 2019; (formerly WorkloadXelerator)Altair FlowTracer™ © 1995- 2019; (formerly FlowTracer)Altair Allocator™ © 1995- 2019; (formerly LicenseAllocator)Altair Monitor™ © 1995- 2019; (formerly LicenseMonitor)Altair Hero™ © 1995- 2019; (formerly HERO)Altair Software Asset Optimization™ (SAO) © 2007- 2019

Note:Compute Manager™ ©2012-2017 is now part of Altair AccessDisplay Manager™ ©2013-2017 is now part of Altair AccessPBS Application Services™ ©2008-2017 is now part of Altair AccessPBS Analytics™ ©2008-2017 is now part of Altair ControlPBS Desktop™ ©2008-2012 is now part of Altair Access, specifically Altair Access desktop, which also has Altair Access web and Altair Access mobilee-Compute™ ©2000-2010 was replaced by “Compute Manager” which is now Altair Access

Altair HyperWorks - A Platform for Innovation®Altair AcuConsole™ ©2006-2019Altair AcuSolve™ ©1997-2019Altair ElectroFlo™ ©1992-2019Altair ESAComp™ ©1992-2019Altair Feko™ ©1999-2014 Altair Development S.A. (Pty) Ltd.; ©2014-2019 Altair Engineering Inc.Altair Flux™ ©1983-2019Altair FluxMotor™ ©2017-2019Altair HyperCrash™ ©2001-2019

Special Notice: Pre-release versions of Altair software are provided ‘as is’, without warranty of any kind. Usage of pre-release versions is strictly limited to non-production purposes.

ii

Page 3: Control Administrator's Guide - Altair PBS Works · Altair Control 2019.1 Administrator's Guide iv • Altair Embed SE™ ©1989-2019 (formerly solidThinking Embed® SE) • Altair

Altair Control 2019.1 Administrator's Guide

Altair HyperGraph™ ©1995-2019Altair HyperMesh™ ©1990-2019Altair HyperStudy™ ©1999-2019Altair HyperView™ ©1999-2019Altair Virtual Wind Tunnel™ ©2012-2019Altair HyperXtrude™ ©1999-2019Altair MotionSolve™ ©2002-2019Altair MotionView™ ©1993-2019Altair Multiscale Designer™ ©2011-2019Altair OptiStruct™ ©1996-2019Altair Radioss™ ©1986-2019Altair SimLab™ ©2004-2019Altair SimSolid™ ©2015-2019Altair nanoFluidX™ © 2013-2018 Fluidyna GmbH, © 2018-2019 Altair Engineering Inc. Altair ultraFluidX™ © 2010-2018 Fluidyna GmbH, © 2018-2019 Altair Engineering Inc. Altair WinProp™ ©2000-2019Altair ConnectMe™ ©2014-2019Plus other products from the Altair solidThinking Platform.

Altair Packaged Solution Offerings (PSOs)Altair Automated Reporting Director™ ©2008-2019Altair GeoMechanics Director™ ©2011-2019Altair Impact Simulation Director™ ©2010-2019Altair Model Mesher Director™ ©2010-2019Altair NVH Director™ ©2010-2019Altair Squeak and Rattle Director™ ©2012-2019Altair Virtual Gauge Director™ ©2012-2019Altair Weight Analytics™ ©2013-2019Altair Weld Certification Director™ ©2014-2019Altair Multi-Disciplinary Optimization Director™ ©2012-2019

solidThinking - Where Innovation Begins™Altair Inspire™ ©2009-2019 including Altair Inspire Motion and Altair Inspire Structures Altair Inspire™ Extrude-Metal ©1996-2019 (formerly Click2Extrude®-Metal) Altair Inspire™ Extrude-Polymer ©1996-2019 (formerly Click2Extrude®-Polymer)Altair Inspire™ Cast ©2011-2019 (formerly Click2Cast®) Altair Inspire™ Form ©1998-2019 (formerly Click2Form®)Altair Inspire™ Mold ©2009-2019Altair Inspire™ Studio ©1993-2019 (formerly ’Evolve’)Altair Compose™ ©2007-2019 (formerly solidThinking Compose®) Altair Activate™ ©1989-2019 (formerly solidThinking Activate®)Altair Embed™ ©1989-2019 (formerly solidThinking Embed®)

iii

Page 4: Control Administrator's Guide - Altair PBS Works · Altair Control 2019.1 Administrator's Guide iv • Altair Embed SE™ ©1989-2019 (formerly solidThinking Embed® SE) • Altair

Altair Control 2019.1 Administrator's Guide

• Altair Embed SE™ ©1989-2019 (formerly solidThinking Embed® SE)• Altair Embed/Digital Power Designer ©2012-2019Altair SimLab™ ©2004-2019Altair 365™ ©1994-2019

Altair SmartWorks™ - Innovation Intelligence®Altair SmartCore™ ©2011-2019Altair SmartEdge™ ©2010-2019Altair SmartSight™ ©2014-2019

Altair intellectual property rights are protected under U.S. and international laws and treaties. Additionally, Altair software is protected under patent #6,859,792 and other patents pending. All other marks are the property of their respective owners.

ALTAIR ENGINEERING INC. Proprietary and Confidential. Contains Trade Secret Information.

Not for use or disclosure outside of Altair and its licensed clients. Information contained in Altair software shall not be decompiled, disassembled, “unlocked”, reverse translated, reverse engineered, or publicly displayed or publicly performed in any manner. Usage of the software is only as explicitly permitted in the end user software license agreement. Copyright notice does not imply publication.

Third party software licenses

AcuConsole contains material licensed from Intelligent Light (www.ilight.com) and used by permission.

Software Security Measures:

Altair Engineering Inc. and its subsidiaries and affiliates reserve the right to embed software security mechanisms in the Software for the purpose of detecting the installation and/or use of illegal copies of the Software. The Software may collect and transmit non-proprietary data about those illegal copies. Data collected will not include any customer data created by or used in connection with the Software and will not be provided to any third party, except as may be required by law or legal process or to enforce our rights with respect to the use of any illegal copies of the Software. By using the Software, each user consents to such detection and collection of data, as well as its transmission and use if an illegal copy of the Software is detected. No steps may be taken to avoid or detect the purpose of any such security mechanisms.

iv

Page 5: Control Administrator's Guide - Altair PBS Works · Altair Control 2019.1 Administrator's Guide iv • Altair Embed SE™ ©1989-2019 (formerly solidThinking Embed® SE) • Altair

Altair Control 2019.1 Administrator's Guide

Technical Support

This document is proprietary information of Altair Engineering, Inc.

Location Telephone e-mail

Australia +1 800 174 396 [email protected] +86 21 6117 1666 [email protected] +33 (0)1 4133 0992 [email protected] +49 (0)7031 6208 22 [email protected] +91 80 66 29 4500

+1 800 425 0234 (Toll Free)[email protected]

Italy +39 800 905595 [email protected] +81 3 6225 5821 [email protected] +82 70 4050 9200 [email protected] +91 80 66 29 4500

+1 800 425 0234 (Toll Free)[email protected]

North America +1 248 614 2425 [email protected] +49 7031 6208 22 [email protected] +46 (0) 46 460 2828 [email protected] +91 80 66 29 4500

+1 800 425 0234 (Toll Free)[email protected]

South Africa +27 21 831 1500 [email protected] America +55 11 3884 0414 [email protected] Kingdom +44 (0)1926 468 600 [email protected]

v

Page 6: Control Administrator's Guide - Altair PBS Works · Altair Control 2019.1 Administrator's Guide iv • Altair Embed SE™ ©1989-2019 (formerly solidThinking Embed® SE) • Altair

Contents

1 Overview......................................................................................................................... 7

1.1 Document Conventions.................................................................................................81.2 About Control............................................................................................................. 91.3 What's New.............................................................................................................. 101.4 Supported Platforms, Browsers and Product Configurations............................................. 111.5 Control Components, Services and Ports...................................................................... 131.6 Third-Party Dependencies........................................................................................... 171.7 PBS Works Licensing..................................................................................................19

2 Prepare for an Upgrade............................................................................................ 21

2.1 Back Up Cloud Data.................................................................................................. 222.2 Preserve Analyze Configuration Files............................................................................ 24

2.2.1 Analyze Data Forwarded by the Data Collector................................................... 242.2.2 Save Analyze Configuration Files.......................................................................24

2.3 Uninstall the Previous Version of Control...................................................................... 252.4 Clean Up Installation Artifacts..................................................................................... 26

3 Install Control and Third-Party Software............................................................. 27

3.1 Deployment Options...................................................................................................283.2 Installation Prerequisites.............................................................................................323.3 PBSWorks Service User.............................................................................................. 333.4 Before Installing Control.............................................................................................353.5 Install socat.............................................................................................................. 363.6 Install Docker............................................................................................................37

3.6.1 Installing Docker for RHEL and CentOS............................................................. 373.6.2 Installing Docker for SLES and SUSE.................................................................38

3.7 Install MongoDB........................................................................................................ 393.7.1 Install MongoDB for RHEL and CentOS.............................................................. 393.7.2 Install MongoDB for SLES and SUSE................................................................. 40

3.8 Run the Cloud Prerequisite Script................................................................................ 413.9 Install Control........................................................................................................... 423.10 Install the Analyze Data Collector.............................................................................. 44

4 Complete the Upgrade...............................................................................................46

4.1 Copy Analyze Configuration Files................................................................................. 474.2 Restore the Cloud Data.............................................................................................. 48

1

Page 7: Control Administrator's Guide - Altair PBS Works · Altair Control 2019.1 Administrator's Guide iv • Altair Embed SE™ ©1989-2019 (formerly solidThinking Embed® SE) • Altair

5 Post-Installation Configuration............................................................................... 52

5.1 Specify the Cloud Administrator's Email........................................................................535.2 Configure a SMTP Server for Bursting Quota Alerts........................................................545.3 Configure a SMTP Server for Monitoring Alerts.............................................................. 555.4 Open Ports................................................................................................................565.5 Set up the Analyze Environment................................................................................. 575.6 Configure Analyze......................................................................................................58

5.6.1 Configure Holidays.......................................................................................... 595.6.2 Configure Prime and Non-Prime Hours...............................................................605.6.3 Configure Exit Codes....................................................................................... 615.6.4 Configure Custom Groups................................................................................ 62

5.7 Log into Control........................................................................................................ 635.8 Disable the IE11 Compatibility View Mode.................................................................... 645.9 Establish Access Controls............................................................................................65

5.9.1 Manage Roles................................................................................................. 655.9.2 Manage Users.................................................................................................72

5.10 Add an HPC Cluster................................................................................................. 79

6 Uninstall Control..........................................................................................................82

6.1 Uninstall Control........................................................................................................ 836.2 Uninstall the Data Collector........................................................................................ 84

7 Start and Stop Services............................................................................................ 85

7.1 Service Commands.................................................................................................... 867.2 Start Control............................................................................................................. 877.3 Stop Control............................................................................................................. 887.4 Restart Control.......................................................................................................... 897.5 Determine the Status of all Control Services.................................................................90

8 Analyze Commands.....................................................................................................92

8.1 Configure Nodes........................................................................................................ 938.2 Configure Holidays..................................................................................................... 988.3 Configure Custom Groups......................................................................................... 1048.4 Configure Exit Codes................................................................................................1088.5 Configure the Data Collector..................................................................................... 1128.6 Configure the Parser................................................................................................ 1158.7 Reset the Analyze Dataset........................................................................................ 1188.8 Change the Analyze Database Password..................................................................... 1228.9 Create a Diagnosis File for Troubleshooting Issues....................................................... 125

2

Page 8: Control Administrator's Guide - Altair PBS Works · Altair Control 2019.1 Administrator's Guide iv • Altair Embed SE™ ©1989-2019 (formerly solidThinking Embed® SE) • Altair

9 Cloud Bursting............................................................................................................126

9.1 Cloud Bursting Overview...........................................................................................1279.2 Cloud Bursting Terminology.......................................................................................1299.3 Prerequisites for Cloud Bursting.................................................................................1309.4 Cloud Bursting Startup Script.................................................................................... 1319.5 Configure Amazon Web Service Cloud Bursting............................................................134

9.5.1 Create an Amazon Web Service User Account................................................... 1349.5.2 Add an Amazon Web Service Cloud Account..................................................... 1359.5.3 Create a Virtual Private Cloud Network............................................................ 1369.5.4 Create an Internet Gateway........................................................................... 1379.5.5 Update the VPC Route Table...........................................................................1379.5.6 Create a Virtual Machine................................................................................1389.5.7 Installing and Configuring the PBS MoM on the VM............................................1409.5.8 Create an OS Image..................................................................................... 1419.5.9 Create an AWS Cloud Bursting Scenario...........................................................142

9.6 Configuring Microsoft Azure Cloud Bursting................................................................. 1459.6.1 Registering the Control Application with Azure.................................................. 1459.6.2 Add a Microsoft Azure Cloud Account...............................................................1489.6.3 Creating a Resource Group.............................................................................1499.6.4 Creating a Virtual Network............................................................................. 1499.6.5 Creating a Virtual Machine............................................................................. 1509.6.6 Installing and Configuring the PBS MoM on the VM............................................1529.6.7 Creating an OS Image...................................................................................1539.6.8 Create an Azure Cloud Bursting Scenario......................................................... 153

9.7 Configure Google Cloud Platform Cloud Bursting.......................................................... 1579.7.1 Create a Project............................................................................................1579.7.2 Create a Service Account............................................................................... 1579.7.3 Add a GCP Cloud Account.............................................................................. 1589.7.4 Create a Virtual Private Cloud Network............................................................ 1599.7.5 Creating a Virtual Machine............................................................................. 1609.7.6 Installing and Configuring the PBS MoM on the VM............................................1619.7.7 Create an OS Image..................................................................................... 1629.7.8 Create a GCP Cloud Bursting Scenario............................................................. 163

9.8 Configure Oracle Cloud Platform Cloud Bursting...........................................................1659.8.1 Create an Oracle Cloud User Account.............................................................. 1659.8.2 Generating an SSH Public for the Oracle Cloud User.......................................... 1669.8.3 Obtain the Root Compartment Identifier...........................................................1679.8.4 Obtain the Tenancy Identifier..........................................................................1689.8.5 Add an Oracle Cloud Account......................................................................... 1689.8.6 Create a Virtual Cloud Network.......................................................................1699.8.7 Creating a Virtual Machine............................................................................. 1719.8.8 Installing and Configuring the PBS MoM on the VM............................................1759.8.9 Create an OS Image..................................................................................... 1769.8.10 Create an Oracle Cloud Bursting Scenario.......................................................176

9.9 Configure Orange Cloud Flexible Engine for Cloud Bursting............................................180

3

Page 9: Control Administrator's Guide - Altair PBS Works · Altair Control 2019.1 Administrator's Guide iv • Altair Embed SE™ ©1989-2019 (formerly solidThinking Embed® SE) • Altair

9.9.1 Create an Orange Cloud Flexible Engine User Account........................................1809.9.2 Select a Region.............................................................................................1829.9.3 Add an Orange Cloud Flexible Engine Account...................................................1829.9.4 Check Orange Cloud Flexible Engine Account Service Quota................................ 1839.9.5 Create a Virtual Private Cloud.........................................................................1849.9.6 Creating a Virtual Machine............................................................................. 1859.9.7 Installing and Configuring the PBS MoM on the VM............................................1869.9.8 Create an OS Image..................................................................................... 1879.9.9 Create a Orange Cloud Cloud Bursting Scenario................................................ 188

9.10 Configure HUAWEI Cloud for Cloud Bursting.............................................................. 1919.10.1 Obtain the HUAWEI Cloud Administrator Credentials........................................ 1919.10.2 Add an HUAWEI Cloud Account..................................................................... 1929.10.3 Check HUAWEI Cloud Account Service Quotas.................................................1939.10.4 Create a Virtual Private Cloud.......................................................................1939.10.5 Creating a Virtual Machine............................................................................1949.10.6 Installing and Configuring the PBS MoM on the VM.......................................... 1969.10.7 Create an OS Image....................................................................................1979.10.8 Create an HUAWEI Cloud Bursting Scenario.................................................... 198

9.11 Configure Open Telekom Cloud for Cloud Bursting...................................................... 2019.11.1 Obtain the OTC Administrator Credentials....................................................... 2019.11.2 Add an OTC Account....................................................................................2029.11.3 Check OTC Account Service Quotas............................................................... 2039.11.4 Create a Virtual Private Cloud.......................................................................2039.11.5 Creating a Virtual Machine............................................................................2059.11.6 Installing and Configuring the PBS MoM on the VM.......................................... 2069.11.7 Create an OS Image....................................................................................2079.11.8 Create an OTC Cloud Bursting Scenario..........................................................208

9.14 Configure PBS Professional for Cloud Bursting........................................................... 2139.14.1 Cloud Provider Instance Types...................................................................... 2139.14.2 Configure Your Network................................................................................2149.14.3 Configure Cloud Bursting Custom Resources................................................... 2149.14.4 Set Cloud Limits at Server........................................................................... 2179.14.5 Configure the Cloud Queues......................................................................... 2179.14.6 Install the PCLM CLI and the Cloud Bursting Hook........................................... 2189.14.7 Test Cloud Bursting with the CLI................................................................... 2209.14.8 Configure the Cloud Bursting Hook................................................................ 2229.14.9 Test Cloud Bursting by Submitting a Job........................................................ 227

9.15 Troubleshoot Cloud Bursting.................................................................................... 2289.16 Manage Cloud and On-Premise Jobs......................................................................... 229

9.16.1 Job Distribution Examples and Solutions.........................................................2299.17 Run Cloud Jobs On-Premise Before Bursting.............................................................. 2319.18 Override Instance Type or Image at Job Submission...................................................2339.19 Request InfiniBand Nodes....................................................................................... 2349.20 AWS Spot Pricing................................................................................................... 235

9.20.1 AWS Spot Pricing Overview.......................................................................... 2359.20.2 Choose Spot Instances.................................................................................2389.20.3 Add Scenarios to the Cloud Bursting Hook......................................................239

4

Page 10: Control Administrator's Guide - Altair PBS Works · Altair Control 2019.1 Administrator's Guide iv • Altair Embed SE™ ©1989-2019 (formerly solidThinking Embed® SE) • Altair

9.21 Modify the Bursting Hook........................................................................................2419.21.1 Change a Scenario in the Cloud Bursting Hook................................................2419.21.2 Add a New Scenario to the Cloud Bursting Hook..............................................2419.21.3 Delete a Scenario from the Cloud Bursting Hook............................................. 2439.21.4 Change the PCLM Server..............................................................................243

9.22 Disable Cloud Bursting............................................................................................2459.23 Uninstall PCLM CLI and Cloud Bursting Hook............................................................. 246

10 Upgrade or Reconfigure PBS Professional........................................................247

10.1 Prepare for a PBS Professional Upgrade.................................................................... 24810.2 Reconfigure Bursting After a PBS Professional Upgrade............................................... 249

10.2.1 Create an OS Image for the New PBS MoM.................................................... 24910.2.2 Update Bursting Scenarios with the New Image.............................................. 24910.2.3 Configure PBS Professional for Cloud Bursting................................................. 250

10.3 Start the Data Collectors........................................................................................ 251

11 Advanced Configuration........................................................................................ 252

11.1 Advanced Configuration by Component..................................................................... 25311.2 Advanced Configuration for the Web Interface........................................................... 255

11.2.1 Advanced Configuration for Authentication...................................................... 25511.2.2 Advanced Configuration for Licensing............................................................. 26011.2.3 Configure the Session Timeout......................................................................26111.2.4 Change the Gateway HTTPS Port...................................................................26111.2.5 Switch to the Locally Installed Webhelp..........................................................261

11.3 Advanced Configuration for Monitoring......................................................................26311.3.1 Configure Concurrent Threads for Retrieving Job Details................................... 26311.3.2 Configure the Job and Node Details Refresh Interval........................................ 26411.3.3 Configure the Monitor Session Timeout.......................................................... 26411.3.4 Configure the Monitor Channel Timeout..........................................................265

11.4 Advanced Configuration for Analyze..........................................................................26611.4.1 Change the Location Where the Data Collector Gathers theAccounting Logs.....................................................................................................26611.4.2 Change the Analyze Hostname......................................................................26611.4.3 Reset Analyze and Simulation Data................................................................267

11.5 Advanced Configuration for Cloud.............................................................................26911.5.1 Access Cloud Analytics................................................................................. 26911.5.2 Access the CloudFlow Interface..................................................................... 26911.5.3 Change the Hostname of the Cloud Component...............................................269

12 Troubleshooting.......................................................................................................271

12.1 Logging................................................................................................................. 27212.1.1 Location of Log Files....................................................................................27212.1.2 Web Interface Logging................................................................................. 273

5

Page 11: Control Administrator's Guide - Altair PBS Works · Altair Control 2019.1 Administrator's Guide iv • Altair Embed SE™ ©1989-2019 (formerly solidThinking Embed® SE) • Altair

12.1.3 Configure and Simulator Logging...................................................................27612.1.4 Monitor Logging...........................................................................................27712.1.5 Analyze Logging.......................................................................................... 27912.1.6 Cloud Logging............................................................................................. 291

12.2 Trace an Issue.......................................................................................................29412.2.1 Diagnosis Script......................................................................................... 29412.2.2 Use the Diagnosis Script.............................................................................. 295

12.3 Troubleshoot Adding a Cluster................................................................................. 29612.4 Troubleshoot Port Conflicts...................................................................................... 29812.5 Troubleshoot a Web Page Rendering Issue in IE11..................................................... 29912.6 Troubleshoot an UnknownHostKey Error.................................................................... 30012.7 Troubleshoot Cloud Bursting DNS Errors................................................................... 30112.8 Troubleshoot an Out of Memory Parser Issue.............................................................30212.9 Troubleshoot Simulation Failures on SLES 12 SP2.......................................................30312.10 Troubleshoot Simulation Failures on SUSE/SLES....................................................... 304

A-1 Install Control Without Internet Connectivity............................................... 305

A-1.1 Install socat......................................................................................................... 306A-1.2 Install Docker.......................................................................................................307A-1.3 Install MongoDB................................................................................................... 308A-1.4 Run the Cloud Prerequisite Script........................................................................... 309A-1.5 Install Control...................................................................................................... 310A-1.6 Install Cloud Manually........................................................................................... 312

Install Docker........................................................................................................312Download and Save the Cloud Images..................................................................... 314Load the Cloud Images.......................................................................................... 315Install the Cloud Containers....................................................................................315Link Cloud with the PBS Works License Server.......................................................... 316Start and Initialize Cloud........................................................................................316Create a Service to Start and Stop Cloud.................................................................318Switch to the Locally Installed Webhelp....................................................................319

A-1.7 Install the Analyze Data Collector........................................................................... 320A-1.8 Complete Upgrade and Post-Installation Steps..........................................................322

A-2 Start and Stop Cloud After a Manual Installation......................................... 323

A-2.1 Start Cloud.......................................................................................................... 324A-2.2 Stop Cloud...........................................................................................................325A-2.3 Restart Cloud....................................................................................................... 326A-2.4 Determine the Status of the Cloud Service.............................................................. 327

A-3 Uninstall Cloud after a Manual Installation....................................................328

Index.................................................................................................................................329

6

Page 12: Control Administrator's Guide - Altair PBS Works · Altair Control 2019.1 Administrator's Guide iv • Altair Embed SE™ ©1989-2019 (formerly solidThinking Embed® SE) • Altair

Overview 1

1 Overview

This chapter covers the following:

• 1.1 Document Conventions (p. 8)

• 1.2 About Control (p. 9)

• 1.3 What's New (p. 10)

• 1.4 Supported Platforms, Browsers and Product Configurations (p. 11)

• 1.5 Control Components, Services and Ports (p. 13)

• 1.6 Third-Party Dependencies (p. 17)

• 1.7 PBS Works Licensing (p. 19)

Page 13: Control Administrator's Guide - Altair PBS Works · Altair Control 2019.1 Administrator's Guide iv • Altair Embed SE™ ©1989-2019 (formerly solidThinking Embed® SE) • Altair

Control Administrator's Guide1 Overview p.8

1.1 Document ConventionsCommon typographical conventions for Altair Control™ technical publications.

PC_HOMEThe Control home directory which contains configuration and logging files. Default is:

/var/spool/pbsworks/2019.1/pbscontrol/home

This default location can be overridden during the installation of Control.

PC_EXECThe Control execution directory which contains binaries and scripts. Default is:

/opt/altair/pbsworks/2019.1/pbscontrol/exec

This default location can be overridden during the installation of Control.

PC_INSTALLERThe Control installer directory which contains installer configuration and log files. Default is:

/opt/altair/pbsworks/2019.1/pbscontrol/installer

The location of this directory is based on the location of PC_EXEC.

PBS_HOMEThe location where the PBS Professional daemon/service configuration files, accounting logs, etc.are installed. Default is:

/var/spool/pbs

PBS_EXECThe location where the PBS Professional executable programs are installed. Default is:

/opt/pbs/

Proprietary Information of Altair Engineering

Page 14: Control Administrator's Guide - Altair PBS Works · Altair Control 2019.1 Administrator's Guide iv • Altair Embed SE™ ©1989-2019 (formerly solidThinking Embed® SE) • Altair

Control Administrator's Guide1 Overview p.9

1.2 About ControlHPC Administrator's control center for managing, optimizing and forecasting HPC resources.

Altair’s Control is an easy-to-use web application for monitoring and managing jobs and nodesof an High-Performance Computing (HPC) cluster with advanced analytics to support data-drivenplanning and decision making. Also, administrators can perform what-if analysis for determining themost productive way to scale an HPC system's resources by running simulations and manage cloudappliances.

Features include:

• Single pane of glass: configure, deploy, monitor, burst, manage, troubleshoot, simulate, analyze,tune

• Real-time monitoring: simplify troubleshooting and maintenance

• Reporting: Analytics powered by Envision

• Workload simulator: simulate and optimize infrastructure sizing

• Multi-cloud bursting: burst to any cloud for peak loads

• Modern UX: drag-and-drop simplicity

Proprietary Information of Altair Engineering

Page 15: Control Administrator's Guide - Altair PBS Works · Altair Control 2019.1 Administrator's Guide iv • Altair Embed SE™ ©1989-2019 (formerly solidThinking Embed® SE) • Altair

Control Administrator's Guide1 Overview p.10

1.3 What's NewNew features available with Altair Control™.

Monitor• A refresh button is available to refresh the Dashboard charts data.

• Sorting of jobs has been extended to additional fields.

• Job and node information can be exported to a CSV file.

• Instances of Allocation Manager are accessible to:

◦ view account balances.

◦ credit and debit an account.

◦ view a summary of credits and debits that have been made against an account.

◦ view account transactions.

Configure• The PBS Professional Server and Scheduler parameters are automatically refreshed every 5

minutes and also when changes are made on the PBS Professional Server. You can also refresh thedata manually by clicking a refresh button.

Cloud• Support for poly-cloud cloud bursting.

• Support for AWS spot pricing.

• DNS configuration for cloud bursting is no longer required.

• Cloud account details can now be viewed by clicking on the name of the cloud account.

• The available PBSWorks burst node licenses is now viewable.

• Cloud nodes that have been burst for a bursting scenario are viewable.

• Access to a CloudFlow interface to follow the application workflows in now available through theCloud tab.

• Log file monitoring through a Kibana interface is available through the Cloud tab.

• Docker container monitoring through a Grafana interface is available through the Cloud tab.

Access Control• Access controls are now available under the Configure tab.

• Managers can provide monitoring access to specific clusters for a role.

• Managers can revoke access to a cluster that has been added by a user via the Monitor tab.

Platform• Support for RHEL/CentOS 7.5 and 7.6. has been added.

• The CPU load required for Control has been significantly reduced.

• Support for IE11 has been added.

Proprietary Information of Altair Engineering

Page 16: Control Administrator's Guide - Altair PBS Works · Altair Control 2019.1 Administrator's Guide iv • Altair Embed SE™ ©1989-2019 (formerly solidThinking Embed® SE) • Altair

Control Administrator's Guide1 Overview p.11

1.4 Supported Platforms, Browsers and ProductConfigurationsA list of supported OS platforms and browsers for the 2019.1 release.

Supported Platforms for ControlCentOS 7.2, 7.3, 7.4, 7.5 and 7.6

RHEL 7.2, 7.3, 7.4, 7.5 and 7.6

SLES 12 SP2

OpenSUSE 42.2

Note: Minor versions of the operating systems listed above can be installed. However, theinstaller will issue a warning message indicating that the unsupported operating system maynot perform as expected. SLES 12 SP3 has not yet been tested.

Supported BrowsersThe latest version of the following browsers are supported:

• Chrome

• Firefox

• Safari

• IE11

Note: For IE11 to render the Control web pages correctly, the Compatibility View modein IE11 must be disabled.

Supported Product ConfigurationsThe currently supported product configurations are:

Table 1: Supported Product Configurations

PBS Professional MongoDB Control

18.2.x and 19.2.1 3.4 & 3.6 2019.1

Important: To use the new Cloud bursting features available with Control 2019.1 requiresPBS Professional 19.2.1 to be installed on the PBS Server.

Proprietary Information of Altair Engineering

Page 17: Control Administrator's Guide - Altair PBS Works · Altair Control 2019.1 Administrator's Guide iv • Altair Embed SE™ ©1989-2019 (formerly solidThinking Embed® SE) • Altair

Control Administrator's Guide1 Overview p.12

Hardware RequirementsControl requires a minimum hardware configuration:

Table 2: Hardware Requirements for Control

Hardware Minimum Requirement Recommended

CPU 8 cores 8 cores

Memory (physical) 16 GB 32 GB

Disk space 80 GB 100 GB

Additional CPUs, memory, and disk space may be required depending upon the Control componentsinstalled and the size of your site's HPC cluster. Large environments may need more memory forAnalyze.

Proprietary Information of Altair Engineering

Page 18: Control Administrator's Guide - Altair PBS Works · Altair Control 2019.1 Administrator's Guide iv • Altair Embed SE™ ©1989-2019 (formerly solidThinking Embed® SE) • Altair

Control Administrator's Guide1 Overview p.13

1.5 Control Components, Services and PortsOverview of the components and services available after installation.

Control ComponentsControl consists of a collection of components and their associated services, accessed and invokedvia a common interface. Each of these components can conceivably reside on a separate host; for thepurposes of this release it is recommended to install everything on a single host, except for the AnalyzeData Collector. A Data Collector must be installed either on the head node of each PBS Professionalcomplex, or on a machine that has access to the PBS Professional accounting logs for that complex.

Components and their associated responsibility are:

Web InterfaceProvides client access to the simulation, analytics, configuration, cloud bursting, and monitoringcapabilities of Control. It also provides traffic routing through the gateway/proxy server andmessaging routing utilizing RabbitMQ.

AnalyzeProvides analytics reporting and simulation results. Sub-components:

• Analyze Server - parses accounting logs and node information and provides analytics.

• Data Collector - copies the accounting logs and node information from your site’s PBSProfessional cluster to the machine where the Analyze server is installed.

SimulatorAllows administrators to run simulations and is responsible for storage of simulation data.

MonitorAllows administrators to monitor and manage HPC nodes and jobs, create alerts, and monitor andmanage instances of Allocation Manager.

ConfigureAllows administrators to configure default WLM server, scheduler and security settings, andmanage queues, resources, resource defaults and limits.

Cloud

Provides cloud bursting capabilities so that jobs can be run in the cloud based on demand.

Proprietary Information of Altair Engineering

Page 19: Control Administrator's Guide - Altair PBS Works · Altair Control 2019.1 Administrator's Guide iv • Altair Embed SE™ ©1989-2019 (formerly solidThinking Embed® SE) • Altair

Control Administrator's Guide1 Overview p.14

Services Installed by ComponentA number of services are installed with Control. Below is a diagram of what services are installed bycomponent.

Note: When the Web Interface, Configure and/or Simulator are installed on the samemachine, a single instance of the Framework services are installed and are shared betweenthe components.

Figure 1: Services by Component

Below is a description of each service:

Component Services/Port Description

pcgui.service (9501) Controls the web application.

pbsworks-nginx.service (9500) Gateway service that controlscommunication betweencomponents.

Web Interface

pbswauth.service (9100) Authentication and licensingservice that providesuser authentication andcommunication with the licenseserver.

Proprietary Information of Altair Engineering

Page 20: Control Administrator's Guide - Altair PBS Works · Altair Control 2019.1 Administrator's Guide iv • Altair Embed SE™ ©1989-2019 (formerly solidThinking Embed® SE) • Altair

Control Administrator's Guide1 Overview p.15

Component Services/Port Description

pbsworks-notification.service

• Web Socket (9105)

• HTTP ReST(9106)

Controls UI notifications.

pbsworks-notification-mongod.service (9575) Notifications database service.

pbsworks-messagerouter-rabbitmq.service(9840)

RabbitMQ service.

pbsworks-messagerouter-http.service (9820) Message router service (forAMQP clients).

pbsworks-messagerouter-mongod.service(9835)

Message router databaseservice.

pbswadmin-orchestrator.service port (9850) Framework HTTP Orchestratorservice.

pbswadmin-worker.service (no port) Framework AMQP Executorservice.

pbswadmin-mongod.service (9860) Framework database service.

pbsmonitor.service (9600) Controls job and nodemonitoring and alerts.

Monitor

pbsmonitor-postgres.service (9603) Monitor database service.

pbswadmin-orchestrator.service (9850) Framework HTTP Orchestratorservice.

Simulator

pbswadmin-worker.service Framework AMQP Executorservice.

pbswadmin-orchestrator.service (9850) Framework HTTP Orchestratorservice.

Configure

pbswadmin-worker.service Framework AMQP Executorservice.

Analyze pbsworks-pbsa.service (9000)

• Tomcat-8 (9080)

• On Demand Analysis HTTP (9143)

• MonetDB (9200)

• Envision HTTPS (9443)

• Envision ActiveMQ (9672)

Provides all back-end analytics.

Proprietary Information of Altair Engineering

Page 21: Control Administrator's Guide - Altair PBS Works · Altair Control 2019.1 Administrator's Guide iv • Altair Embed SE™ ©1989-2019 (formerly solidThinking Embed® SE) • Altair

Control Administrator's Guide1 Overview p.16

Component Services/Port Description

• MongoDB (9700)

pbsworks-dc.service (9343) Analyze Data Collector

Cloud pbscloud.service (9980) Cloud user interface

Open PortsThe following ports need to be opened for communication with external locations and should be open onyour firewall to allow for this communication:

• 9443 - Analyze Data Collector

• 9500 - Control Gateway

• 9980 - Cloud UI

If you are implementing a distributed deployment where Control components are distributed acrossseveral machines, the following ports must also be opened between Control machines.

• 9820 - Message router service (for AMQP clients)

• 9835 - Message router database service

• 9840 - RabbitMQ service

• 9850 - Framework HTTP Orchestrator service

• 9860 - Framework database service

Proprietary Information of Altair Engineering

Page 22: Control Administrator's Guide - Altair PBS Works · Altair Control 2019.1 Administrator's Guide iv • Altair Embed SE™ ©1989-2019 (formerly solidThinking Embed® SE) • Altair

Control Administrator's Guide1 Overview p.17

1.6 Third-Party DependenciesOverview of the third-party software that is required for installing and running Control.

Third-Party Software Requiring InstallationThe following third-party software must be installed manually before installing Control:

DockerDocker must be manually installed prior to installing the Cloud component. The Control installerperforms a check for its presence and the installation of Control is halted and an error message isdisplayed when Docker cannot be found.

MongoDB 3.4 and 3.6MongoDB is used for data storage by all components except Monitor. Due to legal reasons, itsinstallation could not be incorporated into the Control installer for the following components:

• Web Interface

• Analyze

• Configure

• Simulator

MongoDB must be manually installed before installing any of the above components. The Controlinstaller performs a check for its presence and the installation of Control is halted and an errormessage is displayed when MongoDB cannot be found.

socatsocat must be manually installed prior to installing the Web Interface, prior to installing Configurestand-alone, or prior to installing Simulator stand-alone. The Control installer performs a checkfor its presence and the installation of Control is halted and an error message is displayed whensocat cannot be found.

Third-Party Software Automatically Installed with ControlThe following third-party software is installed automatically during the installation of Control. In someinstances, an existing installation of the third-party software may result in a conflict and halt theinstallation of Control.

ApacheApache is installed automatically when any of the Control components are installed and overlaysthe existing Apache installation if present on the system. However, a problem exists on SLES/OpenSUSE platforms, where an existing installation of the apache2-prefork package causes aconflict. In this case, the installation of Control is halted and the installer will display an errormessage directing the user to uninstall all packages related to Apache.

Erlang or Erlang-ERTSA customized version of Erlang is installed when the Web Interface is installed, when Configureis installed stand-alone, or when Simulator is installed stand-alone. The presence of an existinginstallation of Erlang or Erlang-ERTS on the system causes the installation of Control to be halted.The installer will display an error message directing the user to uninstall the product.

Proprietary Information of Altair Engineering

Page 23: Control Administrator's Guide - Altair PBS Works · Altair Control 2019.1 Administrator's Guide iv • Altair Embed SE™ ©1989-2019 (formerly solidThinking Embed® SE) • Altair

Control Administrator's Guide1 Overview p.18

mod_wsgiA customized version of mod_wsgi is installed automatically when any of the Control componentsare installed. The presence of an existing installation of mod_wsgi on the system causes theinstallation of Control to be halted. The installer will display an error message directing the user touninstall the product.

OpenSSL 1.0.2xOpenSSL 1.0.2.x is used by all components of Control. When the presence of OpenSSL isundetectable on the system where Control is being installed, OpenSSL 1.0.2x is automaticallyinstalled. When the presence of OpenSSL is detected on the system, a check for the correctversion is done. If the version of OpenSSL is not 1.0.2x, the installation of Control is halted andthe installer displays an error message directing the user to uninstall OpenSSL. Otherwise, whenthe correct version of OpenSSL is detected, the installation of OpenSSL is skipped.

PBS Professional and PASThe Simulator component requires PBS Professional and PAS. These products are automaticallyinstalled when the Simulator component is installed. The presence of an existing installation ofPBS Professional or PAS on the system causes the installation of Control to be halted. The installerwill display an error message directing the user to uninstall the product.

PostgresPostgres is used by the Monitor component for data storage. It is automatically installed when theMonitor component is installed and by default uses port 9603. As long as an existing installationof Postgres is using a different port, then there should be no conflicts with the instance installedwith Control.

RabbitMQA customized version of RabbitMQ is installed when the Web Interface is installed, when Configureis installed stand-alone, or when Simulator is installed stand-alone. The presence of an existinginstallation of RabbitMQ on the system causes the installation of Control to be halted. The installerwill display an error message directing the user to uninstall the product.

Other installation Considerations:SELinux

SELinux is not a prerequisite for installing Control. However, if it is installed on the machine whereany of the Control components are being installed, it must be disabled.

Proprietary Information of Altair Engineering

Page 24: Control Administrator's Guide - Altair PBS Works · Altair Control 2019.1 Administrator's Guide iv • Altair Embed SE™ ©1989-2019 (formerly solidThinking Embed® SE) • Altair

Control Administrator's Guide1 Overview p.19

1.7 PBS Works LicensingLicensing model for Control.

Licenses must be installed and available via a LMX version 14.0.1 or newer license server. Three typesof licenses are consumed by Control:

• PBSWorks User license - consumed when a user logs into Control

• PBSWorks Node license - consumed when an HPC cluster is added via the Monitor tab or whenan active node is parsed by Analyze when node information (pbsnodes -av) is forwarded by theAnalyze Data Collector.

• PBSWorks Burst Node license - consumed when a node is burst into the cloud.

Licensing for Logging into ControlEach user that logs into Control consumes a PBS Works User license. If a user logs into Control andthen opens a second browser window to access Control, a single PBS Works User license is consumed.

To login to Control at least one PBS Works User license and one PBS Works Node license must beinstalled and available via the Altair LMX license server.

Monitor LicensingWhen a cluster is added to Control, a PBS Works Node license is consumed for each HPC cluster node.A node in Control is considered a PBS Professional vnode regardless of the number of cores or devicesassociated with the vnode.

Once the number of PBS Works Node licenses is exceeded, job and node summary and detailinformation is not available for any newly added HPC clusters. A next to the name of the cluster onthe Job Summary or Node Summary page indicates that Node licenses have been exceeded.

Figure 2: PBS Works Node Licenses Exceeded

Configure LicensingNo licenses are consumed when accessing the capabilities of the Configure component.

Analyze LicensingThe Analyze Data Collector forwards node information (pbsnodes -av) to the Analyze component. Eachactive node consumes a PBSWorks Node license. The total count of PBSWorks Nodes units needed forthe Analyze component levels against the total count of PBSWorks Nodes units needed for the Monitorcomponent.

Proprietary Information of Altair Engineering

Page 25: Control Administrator's Guide - Altair PBS Works · Altair Control 2019.1 Administrator's Guide iv • Altair Embed SE™ ©1989-2019 (formerly solidThinking Embed® SE) • Altair

Control Administrator's Guide1 Overview p.20

Simulator LicensingNo licenses are consumed when accessing the capabilities of the Simulator component.

Cloud LicensingNo licenses are consumed when accessing the capabilities of the Cloud component, except for cloudbursting. PBS Works Burst Node Hours are consumed when nodes are burst in the cloud. For example, if10 nodes are burst for 10 hours, then 100 Burst Node Hours are consumed.

Proprietary Information of Altair Engineering

Page 26: Control Administrator's Guide - Altair PBS Works · Altair Control 2019.1 Administrator's Guide iv • Altair Embed SE™ ©1989-2019 (formerly solidThinking Embed® SE) • Altair

Prepare for an Upgrade 2

2 Prepare for an Upgrade

Instructions for upgrading from a previous version of Control.

This chapter covers the following:

• 2.1 Back Up Cloud Data (p. 22)

• 2.2 Preserve Analyze Configuration Files (p. 24)

• 2.3 Uninstall the Previous Version of Control (p. 25)

• 2.4 Clean Up Installation Artifacts (p. 26)

Skip this chapter if your site is installing Control for the first time. Otherwise, complete these steps andthen install the new version of Control as described in Install Control and Third-Party Software.

Page 27: Control Administrator's Guide - Altair PBS Works · Altair Control 2019.1 Administrator's Guide iv • Altair Embed SE™ ©1989-2019 (formerly solidThinking Embed® SE) • Altair

Control Administrator's Guide2 Prepare for an Upgrade p.22

2.1 Back Up Cloud DataBack up the Cloud data so that cloud account and bursting scenarios do not have to be recreated onceControl 2019.1 is installed.

The following steps must be done as root or as a user with sudo permissions using the sudo command.

It is recommended to have enough space to host two instances of Cloud at the same time or at least 30GB of space.

Restriction: The following instructions cover a migration from Control 2018.3 to 2019.1 orControl 2018.4 to 2019.1.

1. Login to the machine where Cloud is installed.

2. Find your current Cloud version:docker images | grep ui

The version is the second field.pbscloudio.azurecr.io/ui 2018.4 a6ce3e73b071 2 months ago 581MB

3. Export the currently installed Cloud version number:export CLOUD_OLD_VERSION=CURRENT_VERSION

Where CURRENT_VERSION is the version of Control that is currently installed at your site.

4. Export the Cloud version number that you want to upgrade to:export CLOUD_VERSION=2019.1

5. Export the Control home directory:

• If you installed Cloud using the Control installer and installed in the default location:export PC_HOME=/var/spool/pbsworks/$CLOUD_OLD_VERSION/pbscontrol/home

• If you installed Cloud using the Control installer and installed in a non-default location:export PC_HOME=PC_HOME_DIR

Where PC_HOME_DIR is the location of the Control home directory.

6. Export the current installation location of the Cloud kard.

• If you installed Cloud using the Control installer:export KARD_PATH=$PC_HOME/cloudio/kard

• If you installed Cloud manually:export KARD_PATH=CLOUD_DIR/kard

Replace CLOUD_DIR with the installation location of Cloud.

7. Update MongoDB:

a) Install wget:yum install -y wget

b) Install jq:wget https://github.com/stedolan/jq/releases/download/jq-1.6/jq-linux64 \-O /usr/local/bin/jq && chmod +x /usr/local/bin/jq

Proprietary Information of Altair Engineering

Page 28: Control Administrator's Guide - Altair PBS Works · Altair Control 2019.1 Administrator's Guide iv • Altair Embed SE™ ©1989-2019 (formerly solidThinking Embed® SE) • Altair

Control Administrator's Guide2 Prepare for an Upgrade p.23

c) Download or obtain the MongoDB migration script mongo_migration.sh using your usualAltair support channels.

d) Make the script executable:chmod +x mongo_migration.sh

e) Copy the script to the current kard path:cp mongo_migration.sh $KARD_PATH/current

f) Navigate to the kard directory:cd $KARD_PATH/current

g) Update MongoDB:./mongo_migration.sh mongodb 3.6 $(grep db_mongo meta.yml | \awk '{ print $2 }')

If the script completes successfully, the message Mongo database was successfullyupdated version xxxx, you can update your image is displayed.

8. Launch a container:

• When migrating from Control 2018.3 to 2019.1, execute the command:docker run -ti --rm -v /run/docker.sock:/run/docker.sock \-v $KARD_PATH:/pocker/kard pbscloudio.azurecr.io/pocker:$CLOUD_OLD_VERSION bash

• When migrating from Control 2018.4 to 2019.1, execute the command:docker run -ti --rm -v /run/docker.sock:/run/docker.sock \-v $KARD_PATH:/pkr/kard pbscloudio.azurecr.io/pkr:$CLOUD_OLD_VERSION bash

9. Stop Cloud services:

• When migrating from Control 2018.3 to 2019.1, execute the command:pocker clean

• When migrating from Control 2018.4 to 2019.1, execute the command:pkr clean

10. Exit the container:exit

11. Backup your current Cloud data:cd $KARD_PATH/currenttar czvf ~/cloud_kard.tar.gz *

This will create an archive in the home directory of the current user.

Proprietary Information of Altair Engineering

Page 29: Control Administrator's Guide - Altair PBS Works · Altair Control 2019.1 Administrator's Guide iv • Altair Embed SE™ ©1989-2019 (formerly solidThinking Embed® SE) • Altair

Control Administrator's Guide2 Prepare for an Upgrade p.24

2.2 Preserve Analyze Configuration Files

2.2.1 Analyze Data Forwarded by the Data Collector

Once a Data Collector is installed and started, accounting logs, scheduler logs and node information(output from pbsnodes -av) is automatically forwarded to the Analyze Server. Afterward, any changesto the node information and newly generated accounting and scheduler logs are forwarded daily. Thenode information is used to configure Analyze so that administrators do not have to manually configureAnalyze using the pbsa-config-nodes command. Analyze assumes this node configuration is in effectfor all accounting logs that are forwarded from the HPC Complex, including historical data (accountinglogs generated before the Data Collector was installed and started).

If your site has already installed a previous version of Analyze and has generated node details filesby running the pbsa-config-nodes command, you can preserve those files and restore them afterinstallation of Control.

2.2.2 Save Analyze Configuration FilesPreserve Analyze configuration files from a previously installed version of Control.

Save the Analyze configuration files and use them to configure Analyze after installing Control 2019.1

1. Login to the machine where the previous version of Control is installed.

2. Navigate to PC_HOME/config/services/pbsa/parser/modules/pbsdata/config.

3. Copy the following files to a temporary location such as /tmp:

• all holiday_YYYY.conf

• all custom_group_YYYY.conf

• exit_status.conf

• all nodedetals_YYYY.conf

Proprietary Information of Altair Engineering

Page 30: Control Administrator's Guide - Altair PBS Works · Altair Control 2019.1 Administrator's Guide iv • Altair Embed SE™ ©1989-2019 (formerly solidThinking Embed® SE) • Altair

Control Administrator's Guide2 Prepare for an Upgrade p.25

2.3 Uninstall the Previous Version of ControlUninstall the previous version of Control in preparation of installing Control 2019.1.

The following commands must be done as root or as a user with sudo permissions using the sudocommand.

Important: Use the instructions for uninstalling Control that were documented in theAdministrator's guide that was provided for the version of the product that is beinguninstalled. Ex. If Control 2018.4 is installed at your site, then use the instructions foruninstalling the product in the Control 2018.4 Administrator's Guide.

1. Login to the machine where the previous version of Control is installed.

2. Uninstall Control.

3. Login to the machine where the previous version of the Control Data Collector is installed.

4. Uninstall the Data Collector.

5. Repeat step 3 through step 4 to uninstall all Data Collectors.

6. If you manually installed Cloud because the machine where it was installed did not have internetconnectivity:

a) Login to the machine where the previous version of Cloud is installed.

b) Manually uninstall Cloud.

Proprietary Information of Altair Engineering

Page 31: Control Administrator's Guide - Altair PBS Works · Altair Control 2019.1 Administrator's Guide iv • Altair Embed SE™ ©1989-2019 (formerly solidThinking Embed® SE) • Altair

Control Administrator's Guide2 Prepare for an Upgrade p.26

2.4 Clean Up Installation ArtifactsManually clean up some installation artifacts in the case of an unsuccessful uninstallation.

If you suspect that a previous installation of Control was not completely uninstalled (for example, dueto filesystem or permissions issues), make sure the following are removed:

• Control home directory - PC_HOME

• Control execution directory - PC_EXEC

• /usr/sbin/pbsworks-nginx

• /usr/bin/pbs-sim-agent

Proprietary Information of Altair Engineering

Page 32: Control Administrator's Guide - Altair PBS Works · Altair Control 2019.1 Administrator's Guide iv • Altair Embed SE™ ©1989-2019 (formerly solidThinking Embed® SE) • Altair

Install Control and Third-PartySoftware 3

3 Install Control and Third-Party Software

Install required third-party software and all components of Control using the Control installer.

This chapter covers the following:

• 3.1 Deployment Options (p. 28)

• 3.2 Installation Prerequisites (p. 32)

• 3.3 PBSWorks Service User (p. 33)

• 3.4 Before Installing Control (p. 35)

• 3.5 Install socat (p. 36)

• 3.6 Install Docker (p. 37)

• 3.7 Install MongoDB (p. 39)

• 3.8 Run the Cloud Prerequisite Script (p. 41)

• 3.9 Install Control (p. 42)

• 3.10 Install the Analyze Data Collector (p. 44)

Internet connectivity is required to install Control. If there is no internet connectivity, then follow theseinstructions: Install Control Without Internet Connectivity.

Note: If Docker or MongoDB are already installed from a previous installation of Control,then you may skip the steps for their installation.

Page 33: Control Administrator's Guide - Altair PBS Works · Altair Control 2019.1 Administrator's Guide iv • Altair Embed SE™ ©1989-2019 (formerly solidThinking Embed® SE) • Altair

Control Administrator's Guide3 Install Control and Third-Party Software p.28

3.1 Deployment OptionsOverview of the recommended installation option, distributed deployments, and single componentdeployments.

Control consists of a collection of components and their associated services, accessed and invokedvia a common interface. Each of these components can conceivably reside on a separate host; for thepurposes of this release it is recommended to install everything on a single host, except for the AnalyzeData Collector. A Data Collector must be installed either on the head node of each PBS Professionalcomplex, or on a machine that has access to the PBS Professional accounting logs for that complex.

Figure 3: Control Recommended Deployment Option

Installation Sequence1. Install Docker.

2. Install MongoDB.

3. Install Control.

4. Install the Analyze Data Collector for every PBS Professional complex.

Installation TypesThe Control installer offers two installation types for installing its components:

Proprietary Information of Altair Engineering

Page 34: Control Administrator's Guide - Altair PBS Works · Altair Control 2019.1 Administrator's Guide iv • Altair Embed SE™ ©1989-2019 (formerly solidThinking Embed® SE) • Altair

Control Administrator's Guide3 Install Control and Third-Party Software p.29

Basic

Installs all components in a single installation process. Minimal information is requested whenutilizing this installation type:

• License server

• PBSWorks Service User

• Installation location for the home (PC_HOME) and execution directories (PC_EXEC)

AdvancedUsing this option gives you the ability to pick and choose the components to install. Use thisoption:

• to install the Analyze Data Collector on each machine where the PBS Professional accountinglogs are located (typically the PBS Professional Server).

• to distribute components across multiple machines (a distributed deployment).

• when Cloud must be manually installed because there is no internet connectivity. TheAdvanced option is used to install all components, except for Cloud.

• when you do not want to install all components. For example: your site is only interested inthe Monitoring capabilities of Control. The Advanced option is used to install the Monitor andWeb Interface components.

Note: If you are using the Advanced installation type because you are only interestedin installing a single component, then you must also install the Web Interface alongwith the component.

Both installation types assume that all external dependencies have been satisfied prior to installingControl and that the default ports are free and available for use.

Proprietary Information of Altair Engineering

Page 35: Control Administrator's Guide - Altair PBS Works · Altair Control 2019.1 Administrator's Guide iv • Altair Embed SE™ ©1989-2019 (formerly solidThinking Embed® SE) • Altair

Control Administrator's Guide3 Install Control and Third-Party Software p.30

Component DependenciesThere are some dependencies between the Control components. There is an interdependency betweenthe Web Interface and all other components, such that the Web Interface must be installed for anyof the other components to function properly. Additionally, the Simulator is dependent on Analyze forpresenting simulations results. The following diagram shows an overview of these dependencies:

Figure 4: Component Dependencies

Distributed Deployments

Important: It is recommended to contact your Altair support team to assist with adistributed deployment.

Distributed deployments where components are installed across multiple machines are supported. Inthis situation, Control cannot be installed using the Basic installation type. Instead, each componentor subset of components must be installed using the Advanced installation type. When there is ainterdependency between components, the installer will ask where the dependent components havebeen installed.

Important: When installing components distributively, the same username must bespecified for the PBSWorks Service User during the installation of the components.

Example #1: Install the Web Interface singularly on a machine, while installing the rest of thecomponents on a second machine.

Install steps:

1. Install the Web Interface on machine #1

• The installer asks for the hostname/IP address of the machine(s) where Analyze, Cloud,Monitor, Simulator and Configure are installed.

2. Install all other components on machine #2

• The installer asks for the hostname/IP address of the machine(s) where the Web Interface isinstalled.

Proprietary Information of Altair Engineering

Page 36: Control Administrator's Guide - Altair PBS Works · Altair Control 2019.1 Administrator's Guide iv • Altair Embed SE™ ©1989-2019 (formerly solidThinking Embed® SE) • Altair

Control Administrator's Guide3 Install Control and Third-Party Software p.31

Example #2: Install the Web Interface and Monitor component on a machine, while installing the rest ofthe components on a second machine.

Install steps:

1. Install the Web Interface and Monitor component on machine #1

• The installer asks for the hostname/IP address of the machine(s) where Analyze, Cloud,Simulator and Configure are installed.

2. Install all other components on machine #2

• The installer asks for the hostname/IP address of the machine(s) where the Web Interface isinstalled.

Example #3: Install the Simulator component singularly on a machine, while installing all othercomponents on a second machine.

1. Install the Simulator on machine #1

• The installer asks for the hostname/IP address of the machine(s) where Analyze and the WebInterface are installed.

2. Install all other components on machine #2

• The installer asks for the hostname/IP address of the machine(s) where the Web Interface isinstalled.

Installing A Single ComponentYour site may be interested in the capabilities of a single component such as Monitor, Cloud or Simulate.Again, Control cannot be installed using the Basic installation type. Instead, the component that you areinterested in installing must be installed using the Advanced installation type.

The Web Interface must be installed for any of the other components to function properly. It can eitherbe installed on the same machine as the component or it can be installed distributively.

If you are interested in only the simulation capabilities of Control, Analyze must also be installed, as theSimulator is dependent on Analyze for presenting simulations results.

No Internet ConnectivityFor sites where there is no internet connectivity, Cloud must be manually installed after installing allother components of Control.

See AlsoInstall Control Without Internet Connectivity

Proprietary Information of Altair Engineering

Page 37: Control Administrator's Guide - Altair PBS Works · Altair Control 2019.1 Administrator's Guide iv • Altair Embed SE™ ©1989-2019 (formerly solidThinking Embed® SE) • Altair

Control Administrator's Guide3 Install Control and Third-Party Software p.32

3.2 Installation PrerequisitesPrerequisites that must be satisfied before the installation of Control.

Prerequisites for Installing ControlBefore running the Control installer, the following prerequisites must be met:

• Installation must be done as root or as a user with sudo permissions using the sudo command.

• Installation must be done on a machine running on a supported platform.

• You will be prompted for a license server during the installation of Control, therefore a LMX licenseversion 14.0.1 or newer server must be installed prior to installing Control.

• You will be asked to provide a username that will be the PBSWorks Service User during theinstallation of Control. Review the requirements for this user account before installing Control.

• PBSWorks licenses must be purchased depending upon the Control components installed.

Please review the third-party dependencies before installing Control. Several third-party packages mustbe installed prior to installing Control. Additionally, the presence of some third-party software packageson the system may cause a conflict as a customized version of the package is installed with Control.

See AlsoThird-Party DependenciesPBS Works LicensingSupported Platforms, Browsers and Product ConfigurationsPBSWorks Service User

Proprietary Information of Altair Engineering

Page 38: Control Administrator's Guide - Altair PBS Works · Altair Control 2019.1 Administrator's Guide iv • Altair Embed SE™ ©1989-2019 (formerly solidThinking Embed® SE) • Altair

Control Administrator's Guide3 Install Control and Third-Party Software p.33

3.3 PBSWorks Service UserInformation about the PBSWorks Service User.

During the installation of Control, you are asked to provide a username that will be the PBSWorksService User. All Control services including the database instances will run as this user.

The PBSWorks Service User is automatically given managerial privileges and has unrestricted accessto all features of Control. Users who have login access to the machine where Control is installed, canlogin to Control, however they do not have access to any of the features of Control until the PBSWorksService User grants them access privileges.

Prerequisites for the PBSWorks Service UserThe user account chosen for the PBSWorks Service User must meet the following requirements:

• This user must be an existing user on the machine were Control is being installed.

• The PBSWorks Service User must be able to SSH without a password from and into any machinerunning the Simulator component (e.g. from machine A to machine A).

• When installing components distributively, the same username must be specified for the PBSWorksService User during the installation of the components.

PBSWorks Service User's PAS PasswordPasswordless authentication for the version of PAS installed with Control is currently not available.Consequently, the PBSWorks Service User's PAS password must be set after the installation of Controlas the Simulate component uses the PAS Command Line Interface to submit simulation jobs to the PBSServer installed with Control. The PAS password should be set to the PBSWorks Service User's SSHpassword.

Currently, Control does not support using an external PAS/PBS for running simulation jobs, therefore auser account for the PBSWorks Service User does not need to exist on external PAS/PBS installations.

Connect to an HPC Cluster as the PBSWorks Service UserTo monitor or configure an HPC cluster via Control, you will need to login and connect to an HPC cluster.When a cluster is added, the PBS Server hostname or IP address must be provided and then you canconnect to the HPC by using one of two options:

SSH key-based authenticationWith this option, credentials are not provided. Control assumes that SSH key-basedauthentication, between the machine where the Control services are installed and the machinehosting the PBS Server, has been configured for the PBSWorks Service User. Control attempts toconnect to the machine hosting the PBS Server using the PBSWorks Service User username andSSH key. Currently, RSA, DSA and ECDSA are supported.

CredentialsWith this option, credentials of a valid user on the machine hosting the PBS Server are provided.

Once the cluster is added, subsequent administrative actions are performed as the credentials utilizedwhile adding the cluster. It is recommended to add the cluster using the credentials of someone with

Proprietary Information of Altair Engineering

Page 39: Control Administrator's Guide - Altair PBS Works · Altair Control 2019.1 Administrator's Guide iv • Altair Embed SE™ ©1989-2019 (formerly solidThinking Embed® SE) • Altair

Control Administrator's Guide3 Install Control and Third-Party Software p.34

PBS Manager privileges. Connecting to a cluster using non-managerial credentials will severely limit theadministrative actions that can be performed, however monitoring capabilities will not be affected.

If you intend on using the PBSWorks Service User to connect to an HPC cluster and you want to useSSH key-based authentication, then ensure that SSH key-based authentication is set up betweenControl and the PBS Server. It is recommended, but not required for the PBSWorks Service User to havePBS Manager privileges.

Proprietary Information of Altair Engineering

Page 40: Control Administrator's Guide - Altair PBS Works · Altair Control 2019.1 Administrator's Guide iv • Altair Embed SE™ ©1989-2019 (formerly solidThinking Embed® SE) • Altair

Control Administrator's Guide3 Install Control and Third-Party Software p.35

3.4 Before Installing ControlThings to know before you install.

Access to Control Post-InstallationAll users who have login access to the machine where Control is installed, can login to Control (validNIS/SSH users). The PBSWorks Service User entered during the installation of Control has managerialprivileges, therefore has unrestricted access to all features of Control. After installation, the PBSWorksService User must assign access privileges for all other users via the Access Management menuaccessible from the Configure tab. Add a user to one of the default roles or create a new role and addusers to it.

If a user logs into Control and access privileges have not been established for that user, a messageis displayed indicating that the user does not have access to any of the features of Control and theyshould contact their administrator.

Previous PBS Professional/PAS InstallationControl cannot be installed on a machine that already has PBS Professional or PAS installed. The Controlinstaller installs its own version of PBS Professional and PAS for running simulations. An existing installof PBS Professional or PAS will result in a conflict and halts the install. The installer provides an errormessage, directing the user to uninstall PBS Professional and PAS.

Important: Control should NOT be installed on the PBS Professional Server.

See AlsoEstablish Access Controls

Proprietary Information of Altair Engineering

Page 41: Control Administrator's Guide - Altair PBS Works · Altair Control 2019.1 Administrator's Guide iv • Altair Embed SE™ ©1989-2019 (formerly solidThinking Embed® SE) • Altair

Control Administrator's Guide3 Install Control and Third-Party Software p.36

3.5 Install socatInstall the socat package a third-party software dependency.

Use the following instructions for downloading and installing socat:

• For RHEL/CentOS:sudo yum install socat

• For SLES/SUSE, see https://software.opensuse.org/download.html?project=network%3Autilities&package=socat

Proprietary Information of Altair Engineering

Page 42: Control Administrator's Guide - Altair PBS Works · Altair Control 2019.1 Administrator's Guide iv • Altair Embed SE™ ©1989-2019 (formerly solidThinking Embed® SE) • Altair

Control Administrator's Guide3 Install Control and Third-Party Software p.37

3.6 Install DockerInstall Docker which is used to host Cloud.

3.6.1 Installing Docker for RHEL and CentOSInstall Docker for RHEL and CentOS platforms.

• Installation must be done as root or as a user with sudo permissions using the sudo command.

Note: If you want to run Docker as a non-root user, please see instructions to add thisuser to the Docker group. For more information see Manage Docker as a non-root user.

• For more information about installing Docker CE see Get Docker CE for CentOS.

1. Login to the machine where Control is to be installed.

2. Install required packages.yum install -y yum-utils

3. Setup the docker repository:yum-config-manager --add-repo \https://download.docker.com/linux/centos/docker-ce.repo

4. Install the packets and its dependencies:yum install docker-ce

5. Start Docker.systemctl start dockersystemctl enable docker

Proprietary Information of Altair Engineering

Page 43: Control Administrator's Guide - Altair PBS Works · Altair Control 2019.1 Administrator's Guide iv • Altair Embed SE™ ©1989-2019 (formerly solidThinking Embed® SE) • Altair

Control Administrator's Guide3 Install Control and Third-Party Software p.38

3.6.2 Installing Docker for SLES and SUSEInstall Docker for SLES and SUSE platforms.

• Installation must be done as root or as a user with sudo permissions using the sudo command.

Note: If you want to run Docker as a non-root user, please see instructions to add thisuser to the Docker group. For more information see Manage Docker as a non-root user.

• Docker CE is not supported on SLES, therefore Docker EE must be installed.

1. Login to the machine where Control is to be installed.

2. Install and start the latest version of Docker EE by following these instructions Get Docker EE forSLES.

3. Verify that the file /usr/bin/docker-init exists.

4. If the file does not exist:

a) Download the tarball for the latest version of Docker from the Docker repository:

https://download.docker.com/linux/static/stable/x86_64/

b) Extract the archive to /tmp:tar zxvf docker-18.06.1-ce.tgz -C /tmp

c) Execute the command:cp /tmp/docker/docker-init /usr/bin/

Proprietary Information of Altair Engineering

Page 44: Control Administrator's Guide - Altair PBS Works · Altair Control 2019.1 Administrator's Guide iv • Altair Embed SE™ ©1989-2019 (formerly solidThinking Embed® SE) • Altair

Control Administrator's Guide3 Install Control and Third-Party Software p.39

3.7 Install MongoDBInstall MongoDB for use by the Web Interface, Analyze, Configure, and Simulator components for datastorage.

Restriction: Only MongoDB 3.4 and 3.6 are supported.

Due to legal reasons, the installation of MongoDB could not be incorporated into the 2019.1 installer,therefore it must be manually installed, wherever the Web Interface, Analyze, Configure, and Simulatorcomponents are installed. Once MongoDB is installed, no other configuration is necessary.

If the Monitoring component is being installed stand-alone, then MongoDB does not need to beinstalled.

3.7.1 Install MongoDB for RHEL and CentOSInstall MongoDB for RHEL and CentOS platforms.

Installation must be done as root or as a user with sudo permissions using the sudo command.

1. Login to the machine where Control is going to be installed.

2. Create a file /etc/yum.repos.d/mongodb-org-<ver>.repo.

Where <ver> is the version of MongoDB being installed.

3. Edit the file and choose one of the following options:

• Add the following lines for MongoDB 3.4:[mongodb-org-3.4]name=MongoDB Repositorybaseurl=https://repo.mongodb.org/yum/redhat/$releasever/mongodb-org/3.4/x86_64/gpgcheck=1enabled=1gpgkey=https://www.mongodb.org/static/pgp/server-3.4.asc

• Add the following lines for MongoDB 3.6:[mongodb-org-3.6]name=MongoDB Repositorybaseurl=https://repo.mongodb.org/yum/redhat/$releasever/mongodb-org/3.6/x86_64/gpgcheck=1enabled=1gpgkey=https://www.mongodb.org/static/pgp/server-3.6.asc

4. Save the changes made to the file and exit the editor.

5. Install MongoDB using YUM:yum install -y mongodb-org

Proprietary Information of Altair Engineering

Page 45: Control Administrator's Guide - Altair PBS Works · Altair Control 2019.1 Administrator's Guide iv • Altair Embed SE™ ©1989-2019 (formerly solidThinking Embed® SE) • Altair

Control Administrator's Guide3 Install Control and Third-Party Software p.40

3.7.2 Install MongoDB for SLES and SUSEInstall MongoDB for SLES and SUSE platforms.

Installation must be done as root or as a user with sudo permissions using the sudo command.

1. Login to the machine where Control is going to be installed.

2. Import the MongoDB public key by choosing one of the following options:

• To install MongoDB 3.4, enter the command:rpm --import https://www.mongodb.org/static/pgp/server-3.4.asc

• To install MongoDB 3.6, enter the command:rpm --import https://www.mongodb.org/static/pgp/server-3.6.asc

3. Add the repository so that you can install MongoDB using zypper.

• For MongoDB 3.4, enter the command:zypper addrepo --gpgcheck \"https://repo.mongodb.org/zypper/suse/12/mongodb-org/3.4/x86_64/" mongodb

• For MongoDB 3.6, enter the command:zypper addrepo --gpgcheck \"https://repo.mongodb.org/zypper/suse/12/mongodb-org/3.6/x86_64/" mongodb

4. Install MongoDB:

• For MongoDB 3.4, enter the command:zypper install mongodb-org-3.4.10 \mongodb-org-server-3.4.10 \mongodb-org-shell-3.4.10 \mongodb-org-mongos-3.4.10 \ mongodb-org-tools-3.4.10

• For MongoDB 3.6, enter the command:zypper install mongodb-org-3.6.0 \mongodb-org-server-3.6.0 \mongodb-org-shell-3.6.0 \mongodb-org-mongos-3.6.0 \mongodb-org-tools-3.6.0

Proprietary Information of Altair Engineering

Page 46: Control Administrator's Guide - Altair PBS Works · Altair Control 2019.1 Administrator's Guide iv • Altair Embed SE™ ©1989-2019 (formerly solidThinking Embed® SE) • Altair

Control Administrator's Guide3 Install Control and Third-Party Software p.41

3.8 Run the Cloud Prerequisite ScriptPrerequisite check for Cloud.

The following steps must be done as root or as a user with sudo permissions using the sudo command.

Download and run the Cloud prerequisite script to check the following Cloud prerequisites beforeinstallation:

• Presence of Docker and its version

• Space for the images (/var/lib/docker)

• Connectivity with the Cloud Docker repository

Note: The script will complain that it cannot connect to the Cloud Docker repository if youare installing Cloud manually because the machine where you are installing Control does nothave internet connectivity.

1. Download or obtain the pbscloud_preq.sh script using your usual Altair support channels.

2. Run the script to verify that all prerequisites for installing Cloud have been met:./pbscloud_preq.sh

Proprietary Information of Altair Engineering

Page 47: Control Administrator's Guide - Altair PBS Works · Altair Control 2019.1 Administrator's Guide iv • Altair Embed SE™ ©1989-2019 (formerly solidThinking Embed® SE) • Altair

Control Administrator's Guide3 Install Control and Third-Party Software p.42

3.9 Install ControlInstall all Control components using the Basic installation type.

Review the installation prerequisites before proceeding with the installation.

Internet connectivity is required to install Control. If there is no internet connectivity, then follow theseinstructions: Install Control Without Internet Connectivity.

Data Collectors must be installed on the PBS Server to gather accounting log and node information tocreate HPC cluster charts that are accessible through the Analyze tab of Control. The Data Collectormust be installed on the PBS Server in a separate installation process. See Install the Analyze DataCollector.

Note: Installer log files are available in PC_INSTALLER/logs.

Several components use MongoDB for data storage, therefore there will be multiple database instancesrunning once this installation is complete. Monitor and PBS Professional use Postgres for data storage,therefore there will be two Postgres database instances running after this installation is complete.

Every time Control is installed or uninstalled a file called pbswinstall_op_report is appended withparameter information used during the installation/uninstallation process. The file is created inthe directory /var/spool. The contents of this file may be valuable for diagnosing issues with theinstallation/uninstallation of Control.

1. Login to the machine where Control is to be installed.

2. Download or obtain the installer tarballAltairControl_Installer_2019.1_628_20190413_010457.tar.gz using your usual Altairsupport channels.

3. Navigate to the directory where the installer tarball was copied by changing your workingdirectory.

4. Extract the contents of the tarball:tar -xvf AltairControl_Installer_2019.1_628_20190413_010457.tar.gz

A directory called AltairControl_Installer is created.

5. Navigate to the AltairControl_Installer directory:cd AltairControl_Installer

6. Invoke the installer by running the installation binary:./run_installer

If a broken installation is detected then the following message is displayed:Detected an improperly uninstalled or existing/broken installationExisting Altair Control installation needs to be uninstalled. Would you like to proceed? ['yes' or 'no']

Enter yes and press ENTER. Once the broken installation is completely removed, the installationcontinues.

7. Read the introduction, verify the prerequisites for installation, and press ENTER.

Proprietary Information of Altair Engineering

Page 48: Control Administrator's Guide - Altair PBS Works · Altair Control 2019.1 Administrator's Guide iv • Altair Embed SE™ ©1989-2019 (formerly solidThinking Embed® SE) • Altair

Control Administrator's Guide3 Install Control and Third-Party Software p.43

8. Page through the license agreement by pressing ENTER until you are asked to accept its terms andconditions.

9. Accept the license agreement by entering 1 and pressing ENTER.

10. For Installation type, choose Basic by entering 1 and pressing ENTER.

11. For License Server Details, enter the LMX license server in the format port@hostname and pressENTER.

12. For PBSWorks Service User, enter a username who will be the PBSWorks Service User and pressENTER.

The user must exist on the machine where you are installing Control. All Control services includingthe database instances run as this user.

13. For Exec Directory, enter where you would like to install the execution binaries (PC_EXEC) or pressENTER to use the default location.

14. For Home Directory, enter where you would like to install the configuration files (PC_HOME) orpress ENTER to use the default location.

15. Review the pre-installation summary and then press ENTER to install the product.Message similar to those below are displayed as the installation progresses.Installing...-------------

This may take a while...Unzipping files into home, exec, and installer folders....

Progress is: 0% Web Interface Beginning PBSWorks Shared InstallationProgress is: 12% Web Interface Completed updating PBS Works Shared config files.........All milestones reached!!

16. Once the installation is complete, set the PBSWorks Service User's PAS password by issuing thecommand:su - SERVICE_USER -c "/opt/altair/pas/13.2/pas/pas-commands/bin/pas-passwd"

Where SERVICE_USER is the username of the PBSWorks Service User.

17. Enter the PBSWorks Service User's SSH password.

Proprietary Information of Altair Engineering

Page 49: Control Administrator's Guide - Altair PBS Works · Altair Control 2019.1 Administrator's Guide iv • Altair Embed SE™ ©1989-2019 (formerly solidThinking Embed® SE) • Altair

Control Administrator's Guide3 Install Control and Third-Party Software p.44

3.10 Install the Analyze Data CollectorInstall the Analyze Data Collector on each PBS Professional complex.

• Installation must be done as root or as a user with sudo permissions using the sudo command.

• Installation must be done on a machine running on a supported platform.

• The Data Collector must have access to the machine hosting Analyze.

If your site has clusters with the same fully qualified domain name (FQDN), then the Analyze data isaggregated for both clusters under a single FQDN. To allow Analyze data for clusters having the sameFQDN to be tracked separately, a unique name must be provided while installing each Data Collector.

Note: Installer log files are available in PC_INSTALLER/logs.

Every time Control is installed or uninstalled a file called pbswinstall_op_report is appended withparameter information used during the installation/uninstallation process. The file is created inthe directory /var/spool. The contents of this file may be valuable for diagnosing issues with theinstallation/uninstallation of Control.

1. Login to the PBS Professional Server.

2. Download or obtain the installer tarballAltairControl_Installer_2019.1_628_20190413_010457.tar.gz using your usual Altairsupport channels.

3. Navigate to the directory where the installer tarball was copied by changing your workingdirectory.

4. Extract the contents of the tarball:tar -xvf AltairControl_Installer_2019.1_628_20190413_010457.tar.gz

A directory called AltairControl_Installer is created.

5. Navigate to the AltairControl_Installer directory:cd AltairControl_Installer

6. Invoke the installer by running the installation binary:./run_installer

If a broken installation is detected then the following message is displayed:Detected an improperly uninstalled or existing/broken installationExisting Altair Control installation needs to be uninstalled. Would you like to proceed? ['yes' or 'no']

Enter yes and press ENTER. Once the broken installation is completely removed, the installationcontinues.

7. Read the introduction, verify the prerequisites for installation, and press ENTER.

8. Page through the license agreement by pressing ENTER until you are asked to accept its terms andconditions.

9. Accept the license agreement by entering 1 and pressing ENTER.

10. For Installation type, choose Advanced by entering 2 and pressing ENTER.

Proprietary Information of Altair Engineering

Page 50: Control Administrator's Guide - Altair PBS Works · Altair Control 2019.1 Administrator's Guide iv • Altair Embed SE™ ©1989-2019 (formerly solidThinking Embed® SE) • Altair

Control Administrator's Guide3 Install Control and Third-Party Software p.45

11. For component, choose Analyze by entering 5 and pressing ENTER.

12. Choose to install the Data Collector by entering 2 and pressing ENTER.

13. For Web Interface, enter the hostname of the machine where the Control Web Interface isinstalled and press ENTER.

14. For License Server Details, enter the LMX license server in the format port@hostname and pressENTER.

15. For PBSA Server Host, enter the hostname of the machine where Analyze was installed.

16. For Analyse Data Collector FQDN, enter a unique name (FQDN) for the Data Collector.

17. For PBSWorks Service User, enter a username who will be the PBSWorks Service User and pressENTER.

The user must exist on the machine where you are installing Control. All Control services includingthe database instances run as this user.

18. For Exec Directory, enter where you would like to install the execution binaries (PC_EXEC) or pressENTER to use the default location.

19. For Home Directory, enter where you would like to install the configuration files (PC_HOME) orpress ENTER to use the default location.

20. Review the pre-installation summary and then press ENTER to install the product.Message similar to those below are displayed as the installation progresses.Installing... -------------This may take a while...Unzipping files into home, exec, and installer folders.... Progress is: 0% Analyze Beginning PBSWorks Shared InstallationProgress is: 12% Analyze Completed updating PBS Works Shared config files.........All milestones reached!!

21. Update the system path to include the path to the Analyze Commands:export PATH=$PATH:PC_EXEC/dc/scripts/pbsa-commands/bin

Proprietary Information of Altair Engineering

Page 51: Control Administrator's Guide - Altair PBS Works · Altair Control 2019.1 Administrator's Guide iv • Altair Embed SE™ ©1989-2019 (formerly solidThinking Embed® SE) • Altair

Complete the Upgrade 4

4 Complete the Upgrade

Complete the upgrade by copying the preserved Analyze configuration files and restoring the Clouddata.

This chapter covers the following:

• 4.1 Copy Analyze Configuration Files (p. 47)

• 4.2 Restore the Cloud Data (p. 48)

If your site is installing Control for the first time, skip this chapter and continue to the chapter Post-Installation Configuration. Otherwise, complete the steps in this chapter and then proceed to thechapter Post-Installation Configuration.

Page 52: Control Administrator's Guide - Altair PBS Works · Altair Control 2019.1 Administrator's Guide iv • Altair Embed SE™ ©1989-2019 (formerly solidThinking Embed® SE) • Altair

Control Administrator's Guide4 Complete the Upgrade p.47

4.1 Copy Analyze Configuration FilesCopy the Analyze configuration files that were preserved prior to uninstalling the older version ofControl.

1. Login to the machine hosting Analyze.

2. Copy the Analyze configuration files that were preserved prior to uninstalling the older version ofControl to the directory:

PC_HOME/config/pbsa/services/pbsa/parser/modules/pbsdata/config

Warning: This will overwrite any files generated automatically by Analyze.

Proprietary Information of Altair Engineering

Page 53: Control Administrator's Guide - Altair PBS Works · Altair Control 2019.1 Administrator's Guide iv • Altair Embed SE™ ©1989-2019 (formerly solidThinking Embed® SE) • Altair

Control Administrator's Guide4 Complete the Upgrade p.48

4.2 Restore the Cloud DataRestore the Cloud data after upgrading to Control 2019.1.

Restriction: The following instructions cover a migration from Control 2018.3 to 2019.1 orControl 2018.4 to 2019.1.

1. Export the Cloud version number:export CLOUD_VERSION=2019.1

2. Export the Control home directory:

• If you installed Cloud using the Control installer and installed in the default location:export PC_HOME=/var/spool/pbsworks/$CLOUD_VERSION/pbscontrol/home

• If you installed Cloud using the Control installer and installed in a non-default location:export PC_HOME=PC_HOME_DIR

Where PC_HOME_DIR is the location of the Control home directory.

3. Export the current installation location of the Cloud kard.

• If you installed Cloud using the Control installer:export KARD_PATH=$PC_HOME/cloudio/kard

• If you installed Cloud manually:export KARD_PATH=DIR/kard

Replace DIR with the path where Cloud is installed.

4. Launch a pkr container:docker run -ti --rm -e CLOUD_VERSION=$CLOUD_VERSION \-v /run/docker.sock:/run/docker.sock \-v $KARD_PATH:/pkr/kard \pbscloudio.azurecr.io/pkr:$CLOUD_VERSION bash

5. Stop pkr:pkr clean

6. Exit the container:exit

7. Restore the Cloud archive:

a) Backup the kard:mv $KARD_PATH/$CLOUD_VERSION $KARD_PATH/${CLOUD_VERSION}.backup

b) Make a new kard directory:mkdir $KARD_PATH/$CLOUD_VERSION

c) Navigate to the kard directory:cd $KARD_PATH/$CLOUD_VERSION

d) Restore the cloud data:tar zxvf ~/cloud_kard.tar.gz

8. Relaunch a pkr container:docker run -ti --rm -e CLOUD_VERSION=$CLOUD_VERSION \-v /run/docker.sock:/run/docker.sock \-v $KARD_PATH:/pkr/kard pbscloudio.azurecr.io/pkr:$CLOUD_VERSION bash

Proprietary Information of Altair Engineering

Page 54: Control Administrator's Guide - Altair PBS Works · Altair Control 2019.1 Administrator's Guide iv • Altair Embed SE™ ©1989-2019 (formerly solidThinking Embed® SE) • Altair

Control Administrator's Guide4 Complete the Upgrade p.49

9. Navigate to the pkr kard:cd /pkr/kard/$CLOUD_VERSION

10. Update the kard version:

• When migrating from Control 2018.3 to 2019.1, execute the command:sed -i \-e "s#^project_name:.*#project_name: '$(echo $CLOUD_VERSION | \sed 's/\.//')'#" \-e "s#^src_path:.*#src_path: /pkr/kard/$CLOUD_VERSION/./src#" \-e "s#^tag:.*#tag: '$CLOUD_VERSION'#" \-e "s#^driver:.*#driver:\n name: compose#" \-e "s#^env:.*#env: pbscontrol#" \-e "/^hypercloud:.*/,+2d" \-e "/^ui_url_prefix:.*/d" \-e "\$alicense_server:\n \auth_url: $(grep ^auth_url site_config/pacioli.conf | cut -d "=" -f2-)\n \host: $(grep ^host site_config/pacioli.conf | cut -d "=" -f2-)" \meta.yml

• When migrating from Control 2018.4 to 2019.1, execute the command:sed -i \-e "s#^project_name:.*#project_name: '$(echo $CLOUD_VERSION | \sed 's/\.//')'#" \-e "s#^src_path:.*#src_path: /pkr/kard/$CLOUD_VERSION/./src#" \-e "s#^tag:.*#tag: '$CLOUD_VERSION'#" \-e "s#^env:.*#env: pbscontrol#" \-e "/^ui_url_prefix:.*/d" \meta.yml

11. Load the new kard and rebuild the context:pkr kard load $CLOUD_VERSION

12. Update the kard:pkr kard update

13. Update the Docker context:pkr kard make

14. Start Postgres:pkr start -s postgres

15. Exit pkr container:exit

16. Launch a Mistral container:docker run -ti \--rm -v $KARD_PATH/$CLOUD_VERSION/docker-context/mistral-api/mistral.conf:\/home/mistral/data/mistral.conf:ro \-l postgres --network $(docker inspect postgres \-f "{{.HostConfig.NetworkMode}}") mistral-api:$CLOUD_VERSION bash

17. Fix the version of Mistral:

• When migrating from Control 2018.3 to 2019.1, execute the command:mistral-db-manage --config-file /home/mistral/data/mistral.conf stamp 027

• When migrating from Control 2018.4 to 2019.1, execute the command:mistral-db-manage --config-file /home/mistral/data/mistral.conf stamp 028

For more information see Mistral Upgrade Guide.

18. Perform Mistral update:

Proprietary Information of Altair Engineering

Page 55: Control Administrator's Guide - Altair PBS Works · Altair Control 2019.1 Administrator's Guide iv • Altair Embed SE™ ©1989-2019 (formerly solidThinking Embed® SE) • Altair

Control Administrator's Guide4 Complete the Upgrade p.50

mistral-db-manage --config-file /home/mistral/data/mistral.conf upgrade head

19. Exit Mistral container:exit

20. Relaunch a pkr container:docker run -ti --rm -e CLOUD_VERSION=$CLOUD_VERSION \-v /run/docker.sock:/run/docker.sock \-v $KARD_PATH:/pkr/kard pbscloudio.azurecr.io/pkr:$CLOUD_VERSION bash

21. Stop Cloud services:pkr clean

22. Start pkr:pkr start

23. Exit the container:exit

24. Execute the following commands to update the database:

a) Export the MongoDB password:export mongopwd=$(grep db_mongo meta.yml | awk '{ print $2 }')

b) Launch a MongoDB container and a MongoDB shell:docker exec -ti mongodb mongo --authenticationDatabase admin -u admin \-p $mongopwd

You will be placed in a MongoDB shell.

c) Switch to bootstrapper:use bootstrapper

d) Update the database:db.MIGRATIONS.update( {}, { "applied_migrations" : [ "0001_move_ami_id_to_os_image", "0002_add_fields_use_mass_deployment_and_create_infra" ], "migrating" : false })

If the step completes successfully, you will see the message WriteResult({ "nMatched" :0, "nUpserted" : 0, "nModified" : 0 })

e) Exit the shell:exit

f) Execute the following command:docker exec bootstrapper-api bash -c "python -m pclm_common.dbmigrate \--mongodb-admin-password $mongopwd bootstrapper migrate"

g) Execute the following command:docker exec notification-center bash -c "python -m pclm_common.dbmigrate \--mongodb-admin-password $mongopwd notification-center migrate"

h) Execute the following command:docker exec keeper bash -c "python -m pclm_common.dbmigrate keeper migrate"

i) Execute the following command:

Proprietary Information of Altair Engineering

Page 56: Control Administrator's Guide - Altair PBS Works · Altair Control 2019.1 Administrator's Guide iv • Altair Embed SE™ ©1989-2019 (formerly solidThinking Embed® SE) • Altair

Control Administrator's Guide4 Complete the Upgrade p.51

docker exec guardian bash -c "source /opt/guardian_venv/bin/activate \&& python -m pclm_common.dbmigrate guardian migrate"

Proprietary Information of Altair Engineering

Page 57: Control Administrator's Guide - Altair PBS Works · Altair Control 2019.1 Administrator's Guide iv • Altair Embed SE™ ©1989-2019 (formerly solidThinking Embed® SE) • Altair

Post-Installation Configuration 5

5 Post-Installation Configuration

This chapter covers the following:

• 5.1 Specify the Cloud Administrator's Email (p. 53)

• 5.2 Configure a SMTP Server for Bursting Quota Alerts (p. 54)

• 5.3 Configure a SMTP Server for Monitoring Alerts (p. 55)

• 5.4 Open Ports (p. 56)

• 5.5 Set up the Analyze Environment (p. 57)

• 5.6 Configure Analyze (p. 58)

• 5.7 Log into Control (p. 63)

• 5.8 Disable the IE11 Compatibility View Mode (p. 64)

• 5.9 Establish Access Controls (p. 65)

• 5.10 Add an HPC Cluster (p. 79)

Configuration required to complete the installation of Control.

Page 58: Control Administrator's Guide - Altair PBS Works · Altair Control 2019.1 Administrator's Guide iv • Altair Embed SE™ ©1989-2019 (formerly solidThinking Embed® SE) • Altair

Control Administrator's Guide5 Post-Installation Configuration p.53

5.1 Specify the Cloud Administrator's EmailSpecify the Cloud Administrator's email address so that cloud bursting alerts can be sent to that emailaddress.

Note: The following steps are not necessary when Cloud has been installed manually.During the manual install of Cloud, the cloud administrator's email is set when Cloud isinitialized.

1. Open a supported browser and navigate to http://HOSTNAME:9980/control/.

Where HOSTNAME is the hostname or IP address of the machine where Cloud is installed.

2. Login using the default Cloud Administrator credentials.

username = admin@altair and password = Altair@123

3. Navigate to http://HOSTNAME:9980/control/#/user-profile.

Where HOSTNAME is the hostname or IP address of the machine where Cloud is installed.

4. For E-mail, enter the email address where cloud bursting alerts should be sent.

5. Click Save.

Proprietary Information of Altair Engineering

Page 59: Control Administrator's Guide - Altair PBS Works · Altair Control 2019.1 Administrator's Guide iv • Altair Embed SE™ ©1989-2019 (formerly solidThinking Embed® SE) • Altair

Control Administrator's Guide5 Post-Installation Configuration p.54

5.2 Configure a SMTP Server for Bursting QuotaAlertsSet up a SMTP server so that when a cloud bursting resource alert limit is reached an email is sent tothe Cloud Administrator.

The following steps must be executed as root or as a user with sudo permissions using the sudocommand.

1. Create an environment variable for the Cloud version:export CLOUD_VERSION=2019.1

2. Export the current installation location of the Cloud kard.

• If you installed Cloud using the Control installer and installed in the default location:export KARD_PATH=\/var/spool/pbsworks/$CLOUD_VERSION/pbscontrol/home/cloudio/kard

• If you installed Cloud using the Control installer and installed in a non-default location:export KARD_PATH=DIR/cloudio/kard

Replace DIR with the path to the Control home directory.

• If you installed Cloud manually:export KARD_PATH=DIR/kard

Replace DIR with the path where Cloud is installed.

3. Navigate to the directory $KARD_PATH/current.

4. Create/edit the file meta.yml and add the following lines to the file:smtp: host: SMTP_HOST_URL port: SMTP_PORT login: SMTP_LOGIN password: SMTP_PASSWORD

The SMTP Server port number defaults to 25.

Note: The value of login and password can be left blank if a username and passwordare not necessary to use the SMTP server.

5. Save the file.

6. Launch a pkr container:docker run -ti --network host --rm -e CLOUD_VERSION=$CLOUD_VERSION \-v /run/docker.sock:/run/docker.sock \-v $KARD_PATH:/pkr/kard pbscloudio.azurecr.io/pkr:$CLOUD_VERSION bash

7. Update the kard:pkr kard update

8. Update the Docker context:pkr kard make

9. Exit the container:exit

Proprietary Information of Altair Engineering

Page 60: Control Administrator's Guide - Altair PBS Works · Altair Control 2019.1 Administrator's Guide iv • Altair Embed SE™ ©1989-2019 (formerly solidThinking Embed® SE) • Altair

Control Administrator's Guide5 Post-Installation Configuration p.55

5.3 Configure a SMTP Server for Monitoring AlertsConfigure a SMTP server so that Monitoring email alerts are sent to specified users.

The following steps must be executed as root or as a user with sudo permissions using the sudocommand.

A new feature allows alerts to be sent via email to specified users when a nodes goes offline, CPUutilization drops below or rise above a specified percentage, or when a cluster becomes unavailable.These alerts are configured via the Monitoring tab.

If a Postfix SMTP server is installed on the machine where Control is installed and it is running on port25, then email notifications will be sent for the subscribed alerts.

To use another SMTP server to send the emails, provide the hostname and port of the SMPT server andalso the authentication credentials.

1. Login to the machine hosting Control.

2. Edit the file PC_HOME/config/wsbridge/smtp.json.{ "host": "localhost", "port": 25}

3. Update this file with the appropriate SMTP server information.{ "host": "smpt.gmail.com", "port": 587, "user": "[email protected]", "pass": "password"}

4. Save the file.

5. Restart Control:pbs-control-manager restart

Proprietary Information of Altair Engineering

Page 61: Control Administrator's Guide - Altair PBS Works · Altair Control 2019.1 Administrator's Guide iv • Altair Embed SE™ ©1989-2019 (formerly solidThinking Embed® SE) • Altair

Control Administrator's Guide5 Post-Installation Configuration p.56

5.4 Open PortsOpen ports for communication with external locations.

Some ports need to be opened for communication with external locations and should be open on yourfirewall to allow for this communication.

1. Login to the machine where Cloud is installed.

2. Open the following ports:

• 9980 - Cloud UI

3. Login to the machine where the Web Interface is installed.

4. Open the following port:

• 9500 - Control Gateway

5. Login to the machine where the Data Collector is installed.

6. Open the following port:

• 9343 - Analyze Data Collector

7. If Control has been deployed across several machines (distributed deployment):

a) Login to each machine where Control components have been distributed.

b) Open the following ports:

• 9820 - Message router service (for AMQP clients)

• 9835 - Message router database service

• 9840 - RabbitMQ service

• 9850 - Framework HTTP Orchestrator service

• 9860 - Framework database service

Proprietary Information of Altair Engineering

Page 62: Control Administrator's Guide - Altair PBS Works · Altair Control 2019.1 Administrator's Guide iv • Altair Embed SE™ ©1989-2019 (formerly solidThinking Embed® SE) • Altair

Control Administrator's Guide5 Post-Installation Configuration p.57

5.5 Set up the Analyze EnvironmentReset the Analyze database password.

1. Login to the machine where Analyze is installed.

2. Stop Control:pbs-control-manager stop

3. Update the system path to include the path to the Analyze Commands:export PATH=\$PATH:PC_EXEC/pbsa/scripts/pbsa-commands/bin:PC_EXEC/dc/scripts/pbsa-commands/bin

4. Change the default Analyze database password.pbsa-data-password pbsworks@123 NEW_PASSWORD

Where NEW_PASSWORD is a secure site-specific password.

5. Start Control:pbs-control-manager start

Proprietary Information of Altair Engineering

Page 63: Control Administrator's Guide - Altair PBS Works · Altair Control 2019.1 Administrator's Guide iv • Altair Embed SE™ ©1989-2019 (formerly solidThinking Embed® SE) • Altair

Control Administrator's Guide5 Post-Installation Configuration p.58

5.6 Configure AnalyzeOptionally, configure holidays, prime and non-prime time hours, exit codes, and custom groups.

Configure Analyze with information about your sites holiday schedule, prime and non-prime time hours,exit codes, and custom groups. Providing this information allows richer more meaningful Analyze chartsto be produced. When Analyze configuration is skipped, these defaults are applied:

Table 3: Analyze Configuration Defaults

Configuration Category Default

Holiday New Year’s Day

Prime and Non-Prime Time weekdays For Monday through Friday:

• Prime time begins at 09:00 am

• Non-prime time begins at 5:30 pm

Saturday Day is scheduled as non-prime time

Sunday Day is scheduled as non-prime time

Custom Groups "OTHERS"

Exit Codes Successful 0

Unsuccessful -1 through -20

1 through 271

Proprietary Information of Altair Engineering

Page 64: Control Administrator's Guide - Altair PBS Works · Altair Control 2019.1 Administrator's Guide iv • Altair Embed SE™ ©1989-2019 (formerly solidThinking Embed® SE) • Altair

Control Administrator's Guide5 Post-Installation Configuration p.59

5.6.1 Configure HolidaysConfigure site specific holidays.

By default, Analyze is configured with a single holiday, New Year’s day, for the years 2011 through thecurrent year.

Use the pbsa-config-holidays command to add a holiday:

Note: Default entries may be removed using the pbsa-config-holidays --remove option.

Any changes to the holiday information will be incorporated in the next parsing cycle.

1. Login to the machine hosting Analyze.

2. Execute the command:pbsa-config-holidays {HOLIDAY|WEEKDAY} [options]

Execute the following command to add a holiday that takes place over a single day.

pbsa-config-holidays MemorialDay --start-date=2016-05-25 --enddate=2016-05-25

Execute the following command to add a holiday that takes place over multiple days.

pbsa-config-holidays Thanksgiving --start-date=2016-11-26 --end-date=2016-11-27

Execute the following command to add a holiday that takes place in a previous year.

pbsa-config-holidays Thanksgiving --start-date=2014-11-27 --end-date=2014-11-28

Proprietary Information of Altair Engineering

Page 65: Control Administrator's Guide - Altair PBS Works · Altair Control 2019.1 Administrator's Guide iv • Altair Embed SE™ ©1989-2019 (formerly solidThinking Embed® SE) • Altair

Control Administrator's Guide5 Post-Installation Configuration p.60

5.6.2 Configure Prime and Non-Prime HoursAdd prime and non-prime time hours for a weekday.

By default, Analyze is configured with prime and non-prime time hours for the years 2011 through thecurrent year. Prime times hours for Monday through Friday begin at 09:00 am. Non-prime time hoursbegin at 5:30 pm. For Saturday and Sunday, the entire day is scheduled as non-prime time.

Use the pbsa-config-holidays command to add prime and non-prime hours per WEEKDAY:

Note: Default entries may be removed using the pbsa-config-holidays --remove option.

Any changes to the prime/non-prime time information will be incorporated in the next parsing cycle.

1. Login to the machine hosting Analyze.

2. Execute the command:pbsa-config-holidays {HOLIDAY|WEEKDAY} [options]

Execute the following command to configure prime time to start at 9:00 am and non-prime tostart at 6:00 pm for Monday.

pbsa-config-holidays monday --prime-time=09:00 --non-prime-time=18:00

Execute the following command to configure prime time to start at 8:00 am and non-primetime to start at 5:30 pm for Monday. Prime time start defaults to 8:00 am and non-prime startdefaults to 5:30 pm (17:30).

pbsa-config-holidays monday

Execute the following command to configure prime and non-prime time hours for any weekdays(Monday-Friday) not specified separately. For instance, if prime time and non-prime time havealready been specified for the days Monday and Tuesday, then this command will specify primetime and non-prime time for the days Wednesday through Friday.

pbsa-config-holidays weekdays --prime-time=09:00 --non-prime-time=18:00

Execute the following command to configure prime time to start at 9:00 am and non-prime timeto start at 6:00 pm for weekends (Saturday and Sunday).

pbsa-config-holidays weekends --prime-time=09:00 --non-prime-time=18:00

Execute the following command to configure prime time to start at 9:00 am and non-prime tostart at 6:00 pm for Monday for the year 2014 (a previous year’s configuration).

pbsa-config-holidays monday --prime-time=09:00 --non-prime-time=18:00 --start-date=2014-01-01

Proprietary Information of Altair Engineering

Page 66: Control Administrator's Guide - Altair PBS Works · Altair Control 2019.1 Administrator's Guide iv • Altair Embed SE™ ©1989-2019 (formerly solidThinking Embed® SE) • Altair

Control Administrator's Guide5 Post-Installation Configuration p.61

5.6.3 Configure Exit CodesConfigure site specific exit codes.

Analyze maps exit codes to a reason to track the success or failure of a job. Analyze is configured bydefault with two exit reasons “Successful” and “Unsuccessful” that are mapped to specific exit codes.

Use the pbsa-config-exits command to map an exit code to a reason.

Analyze is configured with a default group “Still-Running”. Do not delete this default exit code.

If changes are made to exit codes after the initial parsing cycle is complete, any parsed data must bediscarded and all PBS Professional accounting logs have to be reparsed.

1. Login to the machine hosting Analyze.

2. Execute the command:pbsa-config-exits {EXIT_CODE[,EXIT_CODE] | EXIT_CODE-EXIT_CODE} [options]

Execute the following command to map an exit code of “1” to the reason “Successful” for theapplication PBSJobApp.

pbsa-config-exits 1 --reason=Successful --application=PBSJobApp

Execute the following command to map a list of exit codes to th default reason “Unspecified” forall applications (default).

pbsa-config-exits 2,143,255

Proprietary Information of Altair Engineering

Page 67: Control Administrator's Guide - Altair PBS Works · Altair Control 2019.1 Administrator's Guide iv • Altair Embed SE™ ©1989-2019 (formerly solidThinking Embed® SE) • Altair

Control Administrator's Guide5 Post-Installation Configuration p.62

5.6.4 Configure Custom GroupsAdd a user to a custom group to track utilization at a project level.

Custom groups are used to track utilization at a project level for sites running versions of PBSProfessional prior to 11.2. When a user is added to a custom group, they are essentially being addedto a project. The preferred method for tracking utilization at a project level is the use of the -P optionwhen submitting a job via the PBS Professional qsub command.

Analyze is configured with a default group “OTHERS” for the years 2011 through the current year. Donot delete this default group.

Use the pbsa-config-groups command to add a user to a custom group.

Once the accounting logs are parsed for the first time, modifications to the current year's informationshould only be done in two situations and do not require any reparsing of accounting logs.

1. A user is assigned to a new project.

Use the pbsa-config-groups command to assign the user to the new project. Set the start date totoday's date or later. The end date can be set to a future date or "now".

2. A user completes an assignment.

Use the pbsa-config-groups command to update the user’s completion date by setting the end date tothe assignment completion date.

If the custom group information is modified for any other reason, then discard any parsed data andreparse all PBS Professional accounting logs.

1. Login to the machine hosting Analyze.

2. Execute the command:pbsa-config-groups USER CUSTOMGROUP [options]

Several engineers may be assigned to a project. To track the total number of jobs executedfor that project, Analyze must be configured so that it knows which users are assigned to thatproject.

The following command configures a user “tsmith” to be assigned to a custom group (i.e.,project) “CFD” from January 22, 2016 to November 23, 2016.

pbsa-config-groups tsmith CFD --start-date=2016-01-22 --end-date=2016-11-23

Proprietary Information of Altair Engineering

Page 68: Control Administrator's Guide - Altair PBS Works · Altair Control 2019.1 Administrator's Guide iv • Altair Embed SE™ ©1989-2019 (formerly solidThinking Embed® SE) • Altair

Control Administrator's Guide5 Post-Installation Configuration p.63

5.7 Log into ControlLogin to Control using the URL https://<hostname>:9500/pc.

All users who have login access to the machine where Control is installed, can login to Control (validNIS/SSH users). The PBSWorks Service User entered during the installation of Control has managerialprivileges, therefore has unrestricted access to all features of Control. After installation, the PBSWorksService User must assign access privileges for all other users via the Access Management menuaccessible from the Configure tab. Add a user to one of the default roles or create a new role and addusers to it.

If a user logs into Control and access privileges have not been established for that user, a messageis displayed indicating that the user does not have access to any of the features of Control and theyshould contact their administrator.

1. Open a supported web browser.

2. Enter the URL https://HOSTNAME:9500/pc.

Where HOSTNAME is the hostname or IP address of the machine where the Web Interfacecomponent is installed.

The Control login screen is displayed.

3. Enter your credentials in the username and password fields.

4. Click Log In.

Proprietary Information of Altair Engineering

Page 69: Control Administrator's Guide - Altair PBS Works · Altair Control 2019.1 Administrator's Guide iv • Altair Embed SE™ ©1989-2019 (formerly solidThinking Embed® SE) • Altair

Control Administrator's Guide5 Post-Installation Configuration p.64

5.8 Disable the IE11 Compatibility View ModeDisable the Compatibility View mode so that IE11 renders the Control web pages correctly.

Any users that login to Control using IE11 must disable the Compatibility View mode.

1. Launch Internet Explorer 11.

2. Click located in the upper right-hand corner of the web page.

3. Click Compatibility View settings.

4. Disable Display intranet sites in Compatibility View.

5. Click Close.

Proprietary Information of Altair Engineering

Page 70: Control Administrator's Guide - Altair PBS Works · Altair Control 2019.1 Administrator's Guide iv • Altair Embed SE™ ©1989-2019 (formerly solidThinking Embed® SE) • Altair

Control Administrator's Guide5 Post-Installation Configuration p.65

5.9 Establish Access ControlsCreate roles and add users to these roles to grant and restrict access to the various features of Control.

All users who have login access to the machine where Control is installed, can login to Control (validNIS/SSH users). The PBSWorks Service User entered during the installation of Control has managerialprivileges, therefore has unrestricted access to all features of Control. After installation, the PBSWorksService User must assign access privileges for all other users via the Access Management menuaccessible from the Configure tab. Add a user to one of the default roles or create a new role and addusers to it.

If a user logs into Control and access privileges have not been established for that user, a messageis displayed indicating that the user does not have access to any of the features of Control and theyshould contact their administrator.

Grant access privileges to the various features of Control by using the new Access Control feature. Youcan either add a user to one of the default roles or you can create a new role and add users to it.

Note: Users must be valid NIS/SSH users.

5.9.1 Manage RolesAdd a role, change a role's privileges, control a role's access to a cluster, or delete a role.

Default Roles and PrivilegesOverview of the default access control roles.

Access Control RolesTwo roles are available by default: Manager and Operator. These roles cannot be deleted and theirassigned privileges cannot be changed, however additional users can be added to these roles andaccess to clusters can be granted or restricted for the Operator role.

ManagerA Manager has the highest level of access privilege. Managers can create cloud accounts andbursting scenarios, can view and act upon jobs and nodes, can configure the HPC workloadmanager, can view and modify analysis charts, can run simulations, and can grant and restrictaccess to the various features of Control. By default, the PBSWorks Service User entered duringthe installation of Control is assigned to the Manager role and cannot be removed from this role.

OperatorOperators can view nodes and jobs information and can view alerts.

PrivilegesBy default, the following privilege levels are available:

Proprietary Information of Altair Engineering

Page 71: Control Administrator's Guide - Altair PBS Works · Altair Control 2019.1 Administrator's Guide iv • Altair Embed SE™ ©1989-2019 (formerly solidThinking Embed® SE) • Altair

Control Administrator's Guide5 Post-Installation Configuration p.66

Cloud AdminThis privilege allows full access to the features available via the Cloud tab, allowing the user tocreate cloud accounts and bursting scenarios.

Monitor ViewerThis privilege allows view-only access to the features available via the Monitor tab, allowing theuser to view jobs and node information and alerts.

Monitor AdminThis privilege allows full access to the features available via the Monitor tab, allowing the user tonot only view jobs and nodes information, but also to act upon the job and nodes. For example,placing a job on hold, deleting a job, or taking a node offline. Alerts can also be created.

Configure AdminThis privilege allows full access to the features available via the Configure tab, allowing the userto change HPC workload manager settings and parameters.

Analyze AdminThis privilege allows full access to the features available via the Analyze tab, allowing the user toview, modify, and create analysis charts.

Simulate AdminThis privilege allows full access to the features available via the Simulate tab, allowing the user torun simulations and review simulation results.

When a user is not given privileges to one of the available components of Control (Cloud, Configure,Monitor, Analyze, Simulate), then the associated tab is not displayed. For example, the Operator roleby default is given only Monitor Viewer privileges. A user added to the Operator role will only have viewaccess to the Monitor tab after logging in. The other tabs are not displayed.

Please note that access to certain features available via the Monitor and Configure tab are alsocontrolled by access controls defined at the PBS Professional level. This is dependent upon thecredentials that are used to add the cluster. Once a cluster is added, subsequent administrative actionsare performed as the credentials utilized while adding the cluster. For example, if the user used toconnect to the HPC cluster has PBS Professional User privileges, then that user will not be able toconfigure the HPC workload manager settings via the Configure tab or take a node offline via theMonitor tab.

Proprietary Information of Altair Engineering

Page 72: Control Administrator's Guide - Altair PBS Works · Altair Control 2019.1 Administrator's Guide iv • Altair Embed SE™ ©1989-2019 (formerly solidThinking Embed® SE) • Altair

Control Administrator's Guide5 Post-Installation Configuration p.67

Add a New RoleCreate a custom role for your site.

Review the default roles and privileges before adding a new role.

Two roles are available by default: Manager and Operator. Add additional roles per your site's needs.

When a role is given Monitor Viewer or Monitor Admin privileges, you have the option to assign clustersto the role. Any users assigned that role will automatically be given Monitoring access to those clusters.

1. Click the Configure tab.

2. Click Roles from the Access Management menu located on the left-hand side of the web page.

3. Click Add Role.A role is created with a default role name and undefined privileges.

4. Click located to the right of the name of the role.

a) For Role Name, enter a name that describes the role.

b) Click OK.

5. Choose the privileges to give to this role by clicking located to the right of the privilege.

Privileges are not mutually exclusive, so you may assign more than one.

a) When Monitor Viewer or Monitor Admin are chosen, optionally click to assign clusters to therole.

Figure 5: Assign Cluster

b) Select a cluster by enabling the checkbox next to its name.

You may select more than one cluster.

c) Click OK.Any users assigned to this role will automatically get access to this cluster to monitor jobsand nodes. The following notification is displayed to any of those users who are currentlylogged into Control: Your view is updated to reflect changes in the CLUSTERprivileges where CLUSTER is the name of the cluster.

6. Click Save.The new role is displayed in the Roles list.

Proprietary Information of Altair Engineering

Page 73: Control Administrator's Guide - Altair PBS Works · Altair Control 2019.1 Administrator's Guide iv • Altair Embed SE™ ©1989-2019 (formerly solidThinking Embed® SE) • Altair

Control Administrator's Guide5 Post-Installation Configuration p.68

Change the Privileges of a RoleAdd or remove access privileges for a role.

Review the default roles and privileges before changing the privileges of a role.

You cannot change the access privileges of the default Manager or Operator roles.

When a role is given Monitor Viewer or Monitor Admin privileges, you have the option to assign clustersto the role. Any users assigned that role will automatically be given Monitoring access to those clusters.

1. Click the Configure tab.

2. Click Roles from the Access Management menu located on the left-hand side of the web page.

3. Click the name of the role.

Figure 6: Edit a Role

4. Choose one of the following options to change the privileges given to the role:

• Click located to the right of the privilege to enable the privilege.

• Click located to the right of the privilege to disable the privilege.

Privileges are not mutually exclusive, so you may assign more than one.

a) When Monitor Viewer or Monitor Admin are chosen, optionally click to assign clusters to therole.

Figure 7: Assign Cluster

b) Select a cluster by enabling the checkbox next to its name.

You may select more than one cluster.

c) Click OK.Any users assigned to this role will automatically get access to this cluster to monitor jobsand nodes. The following notification is displayed to any of those users who are currentlylogged into Control: Your view is updated to reflect changes in the CLUSTERprivileges where CLUSTER is the name of the cluster.

5. Click Save.

Proprietary Information of Altair Engineering

Page 74: Control Administrator's Guide - Altair PBS Works · Altair Control 2019.1 Administrator's Guide iv • Altair Embed SE™ ©1989-2019 (formerly solidThinking Embed® SE) • Altair

Control Administrator's Guide5 Post-Installation Configuration p.69

Change the Name of a RoleRename a role.

1. Click the Configure tab.

2. Click Roles from the Access Management menu located on the left-hand side of the web page.

3. Click the name of the role.

Figure 8: Edit a Role

4. Click located to the right of the name of the role.

a) For Role Name, enter a new role name.

b) Click OK.

5. Click Save.

Proprietary Information of Altair Engineering

Page 75: Control Administrator's Guide - Altair PBS Works · Altair Control 2019.1 Administrator's Guide iv • Altair Embed SE™ ©1989-2019 (formerly solidThinking Embed® SE) • Altair

Control Administrator's Guide5 Post-Installation Configuration p.70

Assign a Cluster to a Monitor Viewer or Admin RoleWhen a role is given Monitor Viewer or Monitor Admin privileges, you have the option to assign clustersto the role. Any users assigned to that role will automatically be given monitoring access to thoseclusters.

1. Click the Configure tab.

2. Click Roles from the Access Management menu located on the left-hand side of the web page.

3. Click the name of a role that has been given Monitor Viewer or Monitor Admin privileges.

Figure 9: Choose a Role

4. Click to assign clusters to the role.

Figure 10: Assign Cluster

5. Select a cluster by enabling the checkbox next to its name.

You may select more than one cluster.

6. Click OK.Any users assigned to this role will automatically get access to this cluster to monitor jobs andnodes. The following notification is displayed to any of those users who are currently logged intoControl: Your view is updated to reflect changes in the CLUSTER privileges whereCLUSTER is the name of the cluster.

7. Click Save.

Proprietary Information of Altair Engineering

Page 76: Control Administrator's Guide - Altair PBS Works · Altair Control 2019.1 Administrator's Guide iv • Altair Embed SE™ ©1989-2019 (formerly solidThinking Embed® SE) • Altair

Control Administrator's Guide5 Post-Installation Configuration p.71

Delete a RoleDelete a role that is no longer needed.

1. Click the Configure tab.

2. Click Roles from the Access Management menu located on the left-hand side of the web page.

3. Select a role by enabling the checkbox next to the role's name.

Tip: Select multiple roles so that they can be deleted in a single click.

Figure 11: Select a Role

4. Click .

5. Click Remove.

Proprietary Information of Altair Engineering

Page 77: Control Administrator's Guide - Altair PBS Works · Altair Control 2019.1 Administrator's Guide iv • Altair Embed SE™ ©1989-2019 (formerly solidThinking Embed® SE) • Altair

Control Administrator's Guide5 Post-Installation Configuration p.72

5.9.2 Manage UsersAdd a user, assign or remove a role from a user, revoke access to a cluster, or delete a user.

PBSWorks Service UserInformation about the PBSWorks Service User.

During the installation of Control, you are asked to provide a username that will be the PBSWorksService User. All Control services including the database instances will run as this user.

The PBSWorks Service User is automatically given managerial privileges and has unrestricted accessto all features of Control. Users who have login access to the machine where Control is installed, canlogin to Control, however they do not have access to any of the features of Control until the PBSWorksService User grants them access privileges.

Prerequisites for the PBSWorks Service UserThe user account chosen for the PBSWorks Service User must meet the following requirements:

• This user must be an existing user on the machine were Control is being installed.

• The PBSWorks Service User must be able to SSH without a password from and into any machinerunning the Simulator component (e.g. from machine A to machine A).

• When installing components distributively, the same username must be specified for the PBSWorksService User during the installation of the components.

PBSWorks Service User's PAS PasswordPasswordless authentication for the version of PAS installed with Control is currently not available.Consequently, the PBSWorks Service User's PAS password must be set after the installation of Controlas the Simulate component uses the PAS Command Line Interface to submit simulation jobs to the PBSServer installed with Control. The PAS password should be set to the PBSWorks Service User's SSHpassword.

Currently, Control does not support using an external PAS/PBS for running simulation jobs, therefore auser account for the PBSWorks Service User does not need to exist on external PAS/PBS installations.

Connect to an HPC Cluster as the PBSWorks Service UserTo monitor or configure an HPC cluster via Control, you will need to login and connect to an HPC cluster.When a cluster is added, the PBS Server hostname or IP address must be provided and then you canconnect to the HPC by using one of two options:

SSH key-based authenticationWith this option, credentials are not provided. Control assumes that SSH key-basedauthentication, between the machine where the Control services are installed and the machinehosting the PBS Server, has been configured for the PBSWorks Service User. Control attempts toconnect to the machine hosting the PBS Server using the PBSWorks Service User username andSSH key. Currently, RSA, DSA and ECDSA are supported.

Proprietary Information of Altair Engineering

Page 78: Control Administrator's Guide - Altair PBS Works · Altair Control 2019.1 Administrator's Guide iv • Altair Embed SE™ ©1989-2019 (formerly solidThinking Embed® SE) • Altair

Control Administrator's Guide5 Post-Installation Configuration p.73

CredentialsWith this option, credentials of a valid user on the machine hosting the PBS Server are provided.

Once the cluster is added, subsequent administrative actions are performed as the credentials utilizedwhile adding the cluster. It is recommended to add the cluster using the credentials of someone withPBS Manager privileges. Connecting to a cluster using non-managerial credentials will severely limit theadministrative actions that can be performed, however monitoring capabilities will not be affected.

If you intend on using the PBSWorks Service User to connect to an HPC cluster and you want to useSSH key-based authentication, then ensure that SSH key-based authentication is set up betweenControl and the PBS Server. It is recommended, but not required for the PBSWorks Service User to havePBS Manager privileges.

Add a UserAdd a user so that the user can access the features of Control.

1. Click the Configure tab.

2. Click Users from the Access Management menu located on the left-hand side of the web page.

3. Click Add User.a) For First Name, enter the first name of the user.

b) For Last Name, enter the last name of the user.

c) For User Name, enter the user's NIS/SSH username.

d) Click Save.

What to do next: Assign roles to the user.

Proprietary Information of Altair Engineering

Page 79: Control Administrator's Guide - Altair PBS Works · Altair Control 2019.1 Administrator's Guide iv • Altair Embed SE™ ©1989-2019 (formerly solidThinking Embed® SE) • Altair

Control Administrator's Guide5 Post-Installation Configuration p.74

Assign a Role to a UserAssign a role to the user to establish user privileges.

1. Click the Configure tab.

2. Click Users from the Access Management menu located on the left-hand side of the web page.

3. Select a user by enabling the checkbox next to the user's name.

Tip: Select multiple users when you want to assign the same roles to multiple users.

Figure 12: Select a User

4. Click Assign Roles.

5. Enable the checkbox next to the Role Name to assign the role to the user.

Tip: More than one role can be assigned to the user.

6. Click OK.

Proprietary Information of Altair Engineering

Page 80: Control Administrator's Guide - Altair PBS Works · Altair Control 2019.1 Administrator's Guide iv • Altair Embed SE™ ©1989-2019 (formerly solidThinking Embed® SE) • Altair

Control Administrator's Guide5 Post-Installation Configuration p.75

Remove a Role from a UserRemove a role from a user to limit access to certain features.

1. Click the Configure tab.

2. Click Users from the Access Management menu located on the left-hand side of the web page.

3. Click the name of the user.

Figure 13: Edit a User

4. Click Edit.5. Click to delete the role.

Figure 14: Delete a Role

6. Click Save.

Proprietary Information of Altair Engineering

Page 81: Control Administrator's Guide - Altair PBS Works · Altair Control 2019.1 Administrator's Guide iv • Altair Embed SE™ ©1989-2019 (formerly solidThinking Embed® SE) • Altair

Control Administrator's Guide5 Post-Installation Configuration p.76

Edit the User DetailsChange the first or last name of a user.

1. Click the Configure tab.

2. Click Users from the Access Management menu located on the left-hand side of the web page.

3. Click the name of the user.

Figure 15: Edit a User

4. Click Edit.5. Change the first or last name of the user.

6. Click Save.

Proprietary Information of Altair Engineering

Page 82: Control Administrator's Guide - Altair PBS Works · Altair Control 2019.1 Administrator's Guide iv • Altair Embed SE™ ©1989-2019 (formerly solidThinking Embed® SE) • Altair

Control Administrator's Guide5 Post-Installation Configuration p.77

Delete a UserDelete a user when the user no longer needs access to Control.

1. Click the Configure tab.

2. Click Users from the Access Management menu located on the left-hand side of the web page.

3. Select a user by enabling the checkbox next to the user's name.

Tip: Select multiple users so that they can be deleted in a single click.

Figure 16: Select a User

4. Click .

5. Click Remove.

Proprietary Information of Altair Engineering

Page 83: Control Administrator's Guide - Altair PBS Works · Altair Control 2019.1 Administrator's Guide iv • Altair Embed SE™ ©1989-2019 (formerly solidThinking Embed® SE) • Altair

Control Administrator's Guide5 Post-Installation Configuration p.78

Revoke User Access to a ClusterRevoke access to a cluster that has been added by the user via the Monitor tab.

Users can add clusters via the Monitor tab to connect to an HPC cluster so that jobs and nodes can bemonitored. A Manager can revoke this access to the cluster. Once the access to the cluster is revoked,the user cannot re-add the cluster via the Monitor tab. If the user attempts to re-add the cluster, oneof the following messages are displayed: Cluster name already exists or Server name/IP namealready exists.

1. Click the Configure tab.

2. Click Users from the Access Management menu located on the left-hand side of the web page.

3. Click the name of the user.

Figure 17: Edit a User

4. Click Edit.5. Click to revoke access to the cluster.

Figure 18: Revoke Access to a Cluster

Access to the cluster is revoked for the user. The following notification is displayed to the user ifthe user is currently logged into Control: Your view is updated to reflect changes in theCLUSTER privileges where CLUSTER is the name of the cluster.

6. Click Save.

Proprietary Information of Altair Engineering

Page 84: Control Administrator's Guide - Altair PBS Works · Altair Control 2019.1 Administrator's Guide iv • Altair Embed SE™ ©1989-2019 (formerly solidThinking Embed® SE) • Altair

Control Administrator's Guide5 Post-Installation Configuration p.79

5.10 Add an HPC ClusterConnect to an HPC cluster so that you can monitor and administer the cluster.

To monitor jobs and nodes via the Monitor tab and configure Workload Manager settings via theConfigure tab, you must first add an HPC cluster. Cluster management for the Monitor services and theConfigure services are mutually exclusive, therefore you must manage the clusters for those servicesindependently - i.e, if you add a cluster via the Monitor tab, that cluster is not available in the Configuretab and vice versa.

The user who adds a cluster via the Monitor tab is granted access to the cluster along with any userswho have been given managerial privileges. No other users have access to that cluster unless aManager grants a non-managerial role access to that cluster.

When a user adds a cluster via the Configure tab, all users are granted access to that cluster.

To access and connect to an HPC cluster, the cluster must first be added to Control. When you add acluster, the hostname or IP address of the machine hosting the PBS Server must be provided, and thenyou can connect to the HPC by one of two options:

1. without providing the credentials of a user.

Note: Control assumes that SSH key-based authentication, between the machinewhere the Control services are installed and the machine hosting the PBS Server,has been configured for the user account that is the PBSWorks Service User (asdefined during the installation of Control). Control attempts to connect to the machinehosting the PBS Server using the PBSWorks Service User username and SSH key. Ifthe username or SSH key cannot be verified, an error message is displayed and thecredentials of a valid user on the machine hosting the PBS Server must be entered(Option 2).

2. by providing the credentials of a valid user on the machine hosting the PBS Server.

Once the cluster is added, subsequent administrative actions are performed as the credentials utilizedwhile adding the cluster. It is recommended to add the cluster using the credentials of someone withPBS Manager privileges. Connecting to a cluster using non-managerial credentials will severely limit theadministrative actions that can be performed, however monitoring capabilities will not be affected.

Note: If the machine hosting the PBS Server has been configured to run SSH on a non-default port (default is 22), additional information is necessary to add the cluster.

1. Choose one of the following options:

• Click the Monitor tab to monitor and manage HPC cluster jobs and nodes.

• Click the Configure tab to configure an HPC by changing Workload Manager settings, queuesand resources.

2. Click Manage Clusters.A dialog box appears where information about the cluster is entered.

Proprietary Information of Altair Engineering

Page 85: Control Administrator's Guide - Altair PBS Works · Altair Control 2019.1 Administrator's Guide iv • Altair Embed SE™ ©1989-2019 (formerly solidThinking Embed® SE) • Altair

Control Administrator's Guide5 Post-Installation Configuration p.80

Figure 19: Manage Clusters

3. For Cluster Name, enter the name of the cluster as it will appear in the Control web application.

The cluster name must be 20 characters or less.

Tip: Use the unique name supplied during the installation of the Analyze DataCollector so that analytics data is tracked separately for PBS Complexes having thesame Fully Qualified Domain Name (FQDN).

4. For Host, enter the cluster's host name by entering its:

• IP address - a four-part number string such as 18.72.0.3.

• Host name - a text string such as hpc.altair.com.

5. Click Add.

Control attempts to add the cluster as the PBSWorks Service User using an SSH key. If theusername or SSH key is not valid or the machine hosting the PBS Server has been configured torun SSH on a non-default port, follow these steps:

a) Click Show advanced options.

Figure 20: Additional Information Required

b) For SSH Port, enter the SSH port that the machine hosting the PBS Server listens on.

Change the SSH port if the machine hosting the PBS Server has been configured to run SSHon a non-default port.

c) For Username and Password, enter the username and password of an authenticated user onthe machine hosting the PBS Server.

Proprietary Information of Altair Engineering

Page 86: Control Administrator's Guide - Altair PBS Works · Altair Control 2019.1 Administrator's Guide iv • Altair Embed SE™ ©1989-2019 (formerly solidThinking Embed® SE) • Altair

Control Administrator's Guide5 Post-Installation Configuration p.81

d) Click Add.

The cluster is added.

6. Choose one of the following options:

• Repeat Steps 3 through 5 to add another cluster.

• Click Close to close the Manage Clusters dialog box.

Proprietary Information of Altair Engineering

Page 87: Control Administrator's Guide - Altair PBS Works · Altair Control 2019.1 Administrator's Guide iv • Altair Embed SE™ ©1989-2019 (formerly solidThinking Embed® SE) • Altair

Uninstall Control 6

6 Uninstall Control

Uninstall all components of Control.

This chapter covers the following:

• 6.1 Uninstall Control (p. 83)

• 6.2 Uninstall the Data Collector (p. 84)

Before uninstalling Control, disable cloud bursting.

Page 88: Control Administrator's Guide - Altair PBS Works · Altair Control 2019.1 Administrator's Guide iv • Altair Embed SE™ ©1989-2019 (formerly solidThinking Embed® SE) • Altair

Control Administrator's Guide6 Uninstall Control p.83

6.1 Uninstall ControlUninstall all Control components, services and third-party packages.

Uninstallation must be done as root or as a user with sudo permissions using the sudo command.

Every time Control is installed or uninstalled a file called pbswinstall_op_report is appended withparameter information used during the installation/uninstallation process. The file is created inthe directory /var/spool. The contents of this file may be valuable for diagnosing issues with theinstallation/uninstallation of Control.

Note: All services are automatically stopped during the uninstallation process.

1. Login to the machine where Control is installed.

2. Navigate to the directory where the installer tarball was copied by changing your workingdirectory.

3. Navigate to the AltairControl_Installer directory:cd AltairControl_Installer

4. Invoke the installer by running the installation binary:./run_installer

The following message is displayed:

Would you like to uninstall? (Yes/No) [default: No]:

If a broken installation is detected then the following message is displayed:Detected an improperly uninstalled or existing/broken installationExisting Altair Control installation needs to be uninstalled. Would you like to proceed? ['yes' or 'no']

Enter yes and press ENTER.

5. Enter Yes to uninstall the product and press ENTER.

It may take several minutes to uninstall Control.

All services, files and third-party packages are removed, except for the pbswinstall_op_report file.

Proprietary Information of Altair Engineering

Page 89: Control Administrator's Guide - Altair PBS Works · Altair Control 2019.1 Administrator's Guide iv • Altair Embed SE™ ©1989-2019 (formerly solidThinking Embed® SE) • Altair

Control Administrator's Guide6 Uninstall Control p.84

6.2 Uninstall the Data CollectorUninstall the Analyze Data Collector.

Uninstallation must be done as root or as a user with sudo permissions using the sudo command.

Every time Control is installed or uninstalled a file called pbswinstall_op_report is appended withparameter information used during the installation/uninstallation process. The file is created inthe directory /var/spool. The contents of this file may be valuable for diagnosing issues with theinstallation/uninstallation of Control.

Note: All services are automatically stopped during the uninstallation process.

1. Login to the machine where the Data Collector is installed.

2. Stop the Data Collector:pbs-control-manager stop

3. Navigate to the directory where the installer tarball was copied by changing your workingdirectory.

4. Navigate to the AltairControl_Installer directory:cd AltairControl_Installer

5. Invoke the installer by running the installation binary:./run_installer

The following message is displayed:

Would you like to uninstall? (Yes/No)[default: No]:

If a broken installation is detected then the following message is displayed:Detected an improperly uninstalled or existing/broken installationExisting Altair Control installation needs to be uninstalled. Would you like to proceed? ['yes' or 'no']

Enter yes and press ENTER.

6. Enter Yes to uninstall the product and press ENTER.

It may take several minutes to uninstall the Data Collector.

All services, files and third-party packages are removed, except for the pbswinstall_op_report

7. Repeat the above steps for all Data Collectors.

Proprietary Information of Altair Engineering

Page 90: Control Administrator's Guide - Altair PBS Works · Altair Control 2019.1 Administrator's Guide iv • Altair Embed SE™ ©1989-2019 (formerly solidThinking Embed® SE) • Altair

Start and Stop Services 7

7 Start and Stop Services

This chapter covers the following:

• 7.1 Service Commands (p. 86)

• 7.2 Start Control (p. 87)

• 7.3 Stop Control (p. 88)

• 7.4 Restart Control (p. 89)

• 7.5 Determine the Status of all Control Services (p. 90)

Page 91: Control Administrator's Guide - Altair PBS Works · Altair Control 2019.1 Administrator's Guide iv • Altair Embed SE™ ©1989-2019 (formerly solidThinking Embed® SE) • Altair

Control Administrator's Guide7 Start and Stop Services p.86

7.1 Service CommandsService commands for starting and stopping Control.

All Control services are stopped and started using a single command. Do not attempt to start, stop orrestart individual services on there own.

When the machine hosting Control reboots, all Control services are started automatically.

Proprietary Information of Altair Engineering

Page 92: Control Administrator's Guide - Altair PBS Works · Altair Control 2019.1 Administrator's Guide iv • Altair Embed SE™ ©1989-2019 (formerly solidThinking Embed® SE) • Altair

Control Administrator's Guide7 Start and Stop Services p.87

7.2 Start ControlStart all Control services.

Starting Control must be done as root or as a user with sudo permissions using the sudo command.

1. Login to the machine where Control is installed.

2. Enter the following command to start Control:pbs-control-manager start

Proprietary Information of Altair Engineering

Page 93: Control Administrator's Guide - Altair PBS Works · Altair Control 2019.1 Administrator's Guide iv • Altair Embed SE™ ©1989-2019 (formerly solidThinking Embed® SE) • Altair

Control Administrator's Guide7 Start and Stop Services p.88

7.3 Stop ControlStop all Control services.

Stopping Control must be done as root or as a user with sudo permissions using the sudo command.

1. Login to the machine where Control is installed.

2. Enter the following command to stop Control:pbs-control-manager stop

Proprietary Information of Altair Engineering

Page 94: Control Administrator's Guide - Altair PBS Works · Altair Control 2019.1 Administrator's Guide iv • Altair Embed SE™ ©1989-2019 (formerly solidThinking Embed® SE) • Altair

Control Administrator's Guide7 Start and Stop Services p.89

7.4 Restart ControlRestart all Control services.

Retarting Control must be done as root or as a user with sudo permissions using the sudo command.

1. Login to the machine where Control is installed.

2. Enter the following command to restart Control:pbs-control-manager restart

Proprietary Information of Altair Engineering

Page 95: Control Administrator's Guide - Altair PBS Works · Altair Control 2019.1 Administrator's Guide iv • Altair Embed SE™ ©1989-2019 (formerly solidThinking Embed® SE) • Altair

Control Administrator's Guide7 Start and Stop Services p.90

7.5 Determine the Status of all Control ServicesDetermine whether a Control service is up or down.

1. Login to the machine where Control is installed.

2. Enter the following command to display the status of each Control service:pbs-control-manager status

Messages similar to the following are displayed:pbscloud.service: PBSCLOUD systemctl-status:loaded active exitedpbsmonitor-postgres.service: PBS Monitor Server systemctl-status:loaded active exitedpbsmonitor.service: PBS Monitor Server systemctl-status:loaded active exitedpbswadmin-mongod.service: PBS Works Administrator MongoDB Daemon systemctl-status:loaded active runningpbswadmin-orchestrator.service: PBS Works Service Framework HTTP API systemctl-status:loaded active runningpbswadmin-worker.service: PBS Works Service Framework worker manager systemctl-status:loaded active runningpbswauth.service: PBS Works Auth Server systemctl-status:loaded active exitedpbsworks-dc.service: LSB: Altair PBSWorks Data Collector systemctl-status:loaded active exitedpbsworks-messagerouter-http.service: Message-Router Jobs HTTP API systemctl-status:loaded active runningpbsworks-messagerouter-mongod.service: PBSWorks Message Router MongoDB Daemon systemctl-status:loaded active runningpbsworks-messagerouter-rabbitmq.service: RabbitMQ Server for use by Jobs Service (MessageRouter) systemctl-status:loaded active runningpbsworks-nginx.service: PBSWorks NGINX HTTP and reverse proxy server systemctl-status:loaded active runningpbsworks-notification-mongod.service: PBS Works Notification Server MongoDB Daemon systemctl-status:loaded active runningpbsworks-notification.service: PBS Works WebSocket Notification Server systemctl-status:loaded active runningpbsworks-pbsa.service: LSB: Altair PBSWorks Login server creates customer login sessions systemctl-status:loaded active exitedpcgui.service: PBS Control GUI Server systemctl-status:loaded active runningService pbswadmin-launch-prep is not loaded

Proprietary Information of Altair Engineering

Page 96: Control Administrator's Guide - Altair PBS Works · Altair Control 2019.1 Administrator's Guide iv • Altair Embed SE™ ©1989-2019 (formerly solidThinking Embed® SE) • Altair

Control Administrator's Guide7 Start and Stop Services p.91

Note: It is normal for the pbswadmin-launch-prep service to be down as it runsonce, just prior to other services coming up. It prepares the environment and getseverything ready for the other services to run and then shuts down.

When a service is not running the following message is displayed:Service <service> is not loaded.

where <service> is the name of the service that is down.

Proprietary Information of Altair Engineering

Page 97: Control Administrator's Guide - Altair PBS Works · Altair Control 2019.1 Administrator's Guide iv • Altair Embed SE™ ©1989-2019 (formerly solidThinking Embed® SE) • Altair

Analyze Commands 8

8 Analyze Commands

This chapter covers the following:

• 8.1 Configure Nodes (p. 93)

• 8.2 Configure Holidays (p. 98)

• 8.3 Configure Custom Groups (p. 104)

• 8.4 Configure Exit Codes (p. 108)

• 8.5 Configure the Data Collector (p. 112)

• 8.6 Configure the Parser (p. 115)

• 8.7 Reset the Analyze Dataset (p. 118)

• 8.8 Change the Analyze Database Password (p. 122)

• 8.9 Create a Diagnosis File for Troubleshooting Issues (p. 125)

Page 98: Control Administrator's Guide - Altair PBS Works · Altair Control 2019.1 Administrator's Guide iv • Altair Embed SE™ ©1989-2019 (formerly solidThinking Embed® SE) • Altair

Control Administrator's Guide8 Analyze Commands p.93

8.1 Configure Nodes

Namepbsa-config-nodes

Syntaxpbsa-config-nodes [ options ] HOSTNAME [ HOSTNAME_FAILOVER ] FQDN FILE

DescriptionConfigure Analyze with node details for the PBS Professional Complex represented by HOSTNAME (asspecified in the PBS Professional accounting logs), the failover server HOSTNAME_FAILOVER, and the FQDN(fully qualified domain name) using the FILE containing the output from the pbsnodes-av command.

Note: If the value of HOSTNAME, HOSTNAME_FAILOVER, or FQDN contains a hyphen (-) or aperiod (.), it must be replaced with an underscore (_).

You can set the dates for when the node configuration is in effect using the options --start-date and--end-date.

This command must be executed as the PBSWorks Service User, root or as a user with sudo privilegesusing the sudo command on the machine where Analyze is installed.

Options--config-dir=DIRECTORY

Specify the DIRECTORY where the output of this command is directed. Defaults to PC_HOME/portal/services/pbsa/parser/modules/pbsdata/config.

--end-date=YYYY-MM-DD|nowDefine the effective end date for this PBS Professional node configuration. Default is now whichdenotes that this configuration is currently in effect.

--furnishing-servers=FUR_SERVER[,FUR_SERVER]Specify a comma separated list of FUR_SERVERs when peering is configured.

--helpDisplay help for the command. This option cannot be used in conjunction with any other options.

--newOverwrite any existing node details configuration.

--showDisplay the current configuration on the screen for review. Use this option in conjunction withthe --start-date option to view a configuration from a previous year. Default is to display thecurrent year’s configuration.

--start-date=YYYY-MM-DDDefine the effective start date for this PBS Professional node configuration. Default is the currentdate.

Proprietary Information of Altair Engineering

Page 99: Control Administrator's Guide - Altair PBS Works · Altair Control 2019.1 Administrator's Guide iv • Altair Embed SE™ ©1989-2019 (formerly solidThinking Embed® SE) • Altair

Control Administrator's Guide8 Analyze Commands p.94

--versionDisplay the command's version number. This option cannot be used in conjunction with any otheroptions.

ExamplesAll examples assume that the PBS Professional command pbsnodes -av has been executed and theoutput from this command has been written to a file called pbsnodes.out:

Configure Current Node Details Using Defaults

Configure node detail information that is effective today and will continue to be in effect goingforward (start date defaults to today and end date defaults to now ).

pbsa-config-nodes hwesuse113-01 hwesuse113-01.ms.com pbsnodes.out

Configure a Previous Year’s Node Details pbsa-config-nodes hwesuse113-01 hwesuse113-01.ms.com pbsnodes.out --start-date=2014-01-01 --end-date=2014-12-31

Configure a Future Year’s Node Details pbsa-config-nodes hwesuse113-01 hwesuse113-01.ms.com pbsnodes.out --start-date=2022-01-01 --end-date=2022-12-31

Configure Node Details for a Site Implementing Peer Scheduling

Note: Peer scheduling requires a list of furnishing servers.

pbsa-config-nodes Bangalore bangalore.ms.com pbsnodes.out --furnishing-servers=Troy,MV

Redirect the Output of the Command to an Alternate Location

Redirect the output of the command to an alternate location allowing the output of thecommand to be saved without overwriting the contents of the default configuration directory.

pbsa-config-nodes hwesuse113-01 hwesuse113-01.ms.com pbsnodes.out --config-dir=/tmp/pbsa-config-dir

Refresh the Node Details

Overwrite the current node details. Use this command when you need to discard the currentconfiguration and start afresh.

pbsa-config-nodes hwesuse113-01 hwesuse113-01.ms.com pbsnodes.out --new

Show Previous Year’s Node Detailspbsa-config-nodes --start-date=2014-01-01 --show

Proprietary Information of Altair Engineering

Page 100: Control Administrator's Guide - Altair PBS Works · Altair Control 2019.1 Administrator's Guide iv • Altair Embed SE™ ©1989-2019 (formerly solidThinking Embed® SE) • Altair

Control Administrator's Guide8 Analyze Commands p.95

Error Messages

Table 4: Error Messages

Message Text Description

The PBSWORKS_COMMAND_HOME orPBSWORKS_SERVER_HOME variable was not set, andI was unable to find or open a pbsworks-pbsa.conffile.

The environment variablesPBSWORKS_COMMAND_HOME andPBSWORKS_SERVER_HOME have not beenexported to the environment. Contactyour systems administrator to set up theenvironment properly.

Error occurred while settingPBSWORKS_COMMAND_HOME orPBSWORKS_SERVER_HOME

The environment variablesPBSWORKS_COMMAND_HOME andPBSWORKS_SERVER_HOME have not beenexported to the environment. Contactyour systems administrator to set up theenvironment properly.

Python version XX is not supported. Supportedversion is 2.4.x or higher.

Please update the Python version to 2.4 orhigher.

Error occurred while get uid for user XXXX. Unable to get user id for current user. Contactyour system administrator to report this error.

Operation not permitted, Please run command asXXXX user or ROOT.

The user does not have permissions to runthis command. Please run the command asthe PBSWorks Service User specified duringinstallation or as root user.

memory unit XXXX is not supported An error occurred while processing thepbsnodes -av output during the conversion ofmemory units to GB. The memory unit did notequal one of the following valid options: b, kb,mb, gb, tb, or pb.

The configuration file is empty. This error may occur after executing thiscommand using the --show option. Theconfiguration file is empty, therefore nothinghas been configured yet.

Invalid furnishing server names FUR_SERVERS must be a comma separated listof furnishing servers.

Please check the pbsnodes -av output file, some ofthe nodes are offline. Following nodes are offline:[‘xxxxxxxxxx’]

Please investigate the reason for the downednode and either correct the issue within PBSProfessional, or if this is a planned outage,

Proprietary Information of Altair Engineering

Page 101: Control Administrator's Guide - Altair PBS Works · Altair Control 2019.1 Administrator's Guide iv • Altair Embed SE™ ©1989-2019 (formerly solidThinking Embed® SE) • Altair

Control Administrator's Guide8 Analyze Commands p.96

Message Text Description

then manually edit the pbsnodes -av outputfile and modify the state of the node to anonline status, or delete the node.

Incorrect number of arguments. Try pbsa-config-nodes --help.

Please review the usage information forthis command. You can obtain the usageinformation by using the --help option.

Config directory does not exists. Path XXXX. The configuration directory PC_HOME/portal/services/pbsa/parser/modules/pbsdata/config does not exist. An error may haveoccurred during installation. Please re-installthe product.

Unable to instantiate the PBSA ToolKit. Please checkyour PBSWORKS_SERVER_HOME variable.

The environment variablePBSWORKS_SERVER_HOME may not havebeen exported to the environment. Contactyour systems administrator to set up theenvironment properly.

Unable to read the "pbsnodes -av" output file for userXXXX.

Path: XXXX.

The pbsnodes -av output file does not exist.

Unable to read the "pbsnodes -av" output file for userXXXX.

Please check file permissions.

Incorrect file permissions for opening thepbsnodes -av output file for read access.

Unable to open the “nodedetails_<year>.conf ” file. Please check the existence and permissionsof the nodedetails_YYYY.conf file locatedat PC_HOME/portal/services/pbsa/parser/modules/pbsdata/config.

The startdate XXXX provided is not valid. Value of --start-date must be in the formatYYYY-MM-DD.

The enddate XXXX provided is not valid. Value of --end-date must be in the formatYYYY-MM-DD or now.

The startdate XXXX provided must not be greaterthan end date XXXX.

Invalid date range. Value of --start-datemust be less than the value of --end-date.

The startdate year XXXX and enddate year XXXXmust be the same.

Specified date range must not span overseveral years.

Proprietary Information of Altair Engineering

Page 102: Control Administrator's Guide - Altair PBS Works · Altair Control 2019.1 Administrator's Guide iv • Altair Embed SE™ ©1989-2019 (formerly solidThinking Embed® SE) • Altair

Control Administrator's Guide8 Analyze Commands p.97

Message Text Description

Start date format is incorrect. Expected is YYYY-MM-DD.

Dates should be in the format YYYY-MM-DD.

Proprietary Information of Altair Engineering

Page 103: Control Administrator's Guide - Altair PBS Works · Altair Control 2019.1 Administrator's Guide iv • Altair Embed SE™ ©1989-2019 (formerly solidThinking Embed® SE) • Altair

Control Administrator's Guide8 Analyze Commands p.98

8.2 Configure Holidays

Namepbsa-config-holidays

Syntaxpbsa-config-holidays [ options ] { HOLIDAY | { monday | tuesday | wednesday | thursday |friday | saturday | sunday | weekdays | weekends } }

DescriptionAdd a HOLIDAY or configure prime and non-prime time hours for a day of the week, or for any weekdaysthat have not already been defined, or for weekends (Saturday and Sunday).

You can set the dates for the holiday using the options --start-date and --end-date.

You can set prime and non-prime time hours using the options --prime-time and --non-prime-time.

By default, a single holiday, New Year’s day, is configured for the years 2011 through the current year.For Monday through Friday, prime times hours begin at 8:00 am, and non-prime time hours begin at5:30 pm. For Saturday and Sunday, the entire day is scheduled as non-prime time.

This command must be executed as the PBSWorks Service User, root or as a user with sudo privilegesusing the sudo command on the machine where Analyze is installed.

Options--config-dir=DIRECTORY

Specify the DIRECTORY where the output of this command is directed. Defaults to PC_HOME/portal/services/pbsa/parser/modules/pbsdata/config.

--end-date=YYYY-MM-DD|nowDefine the end date of the holiday. Default is the current date.

--helpDisplay help for the command. This option cannot be used in conjunction with any other options.

--newOverwrite any existing node details configuration.

--non-prime-time=HH:MM|all|none

Define the non-prime time start time. Default is 17:30.

HH:MM designates the start time of non-prime time hours in a 24 hour format.

all designates that all hours are non-prime time hours.

none designates that all hours are prime time hours.

--prime-time=HH:MM|all|none

Define the prime time start time. Default is 08:00.

HH:MM designates the start time of prime time hours in a 24 hour format.

Proprietary Information of Altair Engineering

Page 104: Control Administrator's Guide - Altair PBS Works · Altair Control 2019.1 Administrator's Guide iv • Altair Embed SE™ ©1989-2019 (formerly solidThinking Embed® SE) • Altair

Control Administrator's Guide8 Analyze Commands p.99

all designates that all hours are prime time hours.

none designates that all hours are non-prime time hours.

--removeRemove a HOLIDAY or prime and non-prime time hours for a day of the week, or for any weekdaysthat have not already been defined, or for weekends.

--showDisplay the current configuration on the screen for review. Use this option in conjunction with the--start-date option to view a configuration from a previous year. Default is to display the currentyear’s configuration.

--start-date=YYYY-MM-DD

Define the start date of the holiday. This option can also be used to configure prime/non-primetime hours for a previous year’s configuration. Default is the current date.

--versionDisplay the command's version number. This option cannot be used in conjunction with any otheroptions.

Examples

Configure a Holiday that Occurs Over a Single Daypbsa-config-holidays MemorialDay --start-date=2016-05-25 --end-date=2016-05-25

Configure a Holiday that Occurs Over Multiple Dayspbsa-config-holidays Thanksgiving --start-date=2016-11-26 --end-date=2016-11-27

Configure Prime Time for 9:00 am and Non-Prime for 6:00 pm on Mondaypbsa-config-holidays monday --prime-time=09:00 --non-prime-time=18:00

Configure Prime and Non-Prime Time Hours for Tuesday Using the Defaultspbsa-config-holidays tuesday

Configure Prime Time for 9 am and Non-Prime Time for 6 pm for the Weekendpbsa-config-holidays weekends --prime-time=09:00 --non-prime-time=18:00

Configure Prime and Non-Prime Hours for Remaining Weekdays

Configure prime and non-prime time hours for any weekdays that have not already beenconfigured separately. For example, if prime and non-prime time hours have already beenconfigured for Monday and Tuesday, then this command will specify prime and non-prime timehours for the weekdays Wednesday through Friday.

pbsa-config-holidays weekdays --prime-time=09:00 --non-prime-time=18:00

Proprietary Information of Altair Engineering

Page 105: Control Administrator's Guide - Altair PBS Works · Altair Control 2019.1 Administrator's Guide iv • Altair Embed SE™ ©1989-2019 (formerly solidThinking Embed® SE) • Altair

Control Administrator's Guide8 Analyze Commands p.100

Configure Prime and Non-prime Hours for Monday for a Previous Year pbsa-config-holidays monday --start-date=2014-01-01 --prime-time=09:00 --non-prime-time=18:00

Remove a Holidaypbsa-config-holidays MemorialDay --remove

Remove Prime and Non-Prime Time Hours for a Specific Daypbsa-config-holidays monday --remove

Redirect the Output of the Command to an Alternate Location

Redirect the output of the command to an alternate location allowing the output of thecommand to be saved without overwriting the contents of the default configuration directory.

pbsa-config-nodes hwesuse113-01 hwesuse113-01.ms.com pbsnodes.out --config-dir=/tmp/pbsa-config-dir

Refresh the Holidays and Prime and Non-Prime Time Hours

Overwrite the current holidays and prime and non-prime time hours. Use this command whenyou need to discard the current configuration and start afresh.

pbsa-config-nodes hwesuse113-01 hwesuse113-01.ms.com pbsnodes.out --new

Show the Holidays and Prime and Non-Prime Time Hours for a Previous Yearpbsa-config-holidays --start-date=2014-01-01 --show

Error Messages

Table 5: Error Messages

Message Text Description

The PBSWORKS_COMMAND_HOME orPBSWORKS_SERVER_HOME variable was not set, andI was unable to find or open a pbsworks-pbsa.conffile.

The environment variablesPBSWORKS_COMMAND_HOME andPBSWORKS_SERVER_HOME have not beenexported to the environment. Contactyour systems administrator to set up theenvironment properly.

Error occurred while settingPBSWORKS_COMMAND_HOME orPBSWORKS_SERVER_HOME

The environment variablesPBSWORKS_COMMAND_HOME andPBSWORKS_SERVER_HOME have not beenexported to the environment. Contact

Proprietary Information of Altair Engineering

Page 106: Control Administrator's Guide - Altair PBS Works · Altair Control 2019.1 Administrator's Guide iv • Altair Embed SE™ ©1989-2019 (formerly solidThinking Embed® SE) • Altair

Control Administrator's Guide8 Analyze Commands p.101

Message Text Description

your systems administrator to set up theenvironment properly.

Python version XX is not supported. Supportedversion is 2.4.x or higher.

Please update the Python version to 2.4 orhigher.

Error occurred while get uid for user XXXX. Unable to get user id for current user. Contactyour system administrator to report this error.

Operation not permitted, Please run command asXXXX user or ROOT.

The user does not have permissions to runthis command. Please run the command asthe PBSWorks Service User specified duringinstallation or as root user.

Either Holiday or prime and non-prime config can bespecified at a time.

Configuring a holiday and prime/non-primetime hours cannot be done with a singlecommand. First issue the command toconfigure the holidays, and then issue a secondcommand to configure the prime/non-primetime hours.

The configuration file is empty. This error may occur after executing thiscommand using the --show option. Theconfiguration file is empty, therefore nothinghas been configured yet.

Invalid holiday name. Holiday name must containonly any alphanumeric character and the underscore.

Please enter a valid holiday name whichcontains only alphanumeric characters and/orthe underscore character.

Invalid option --remove in the command. Cannot specify the --remove option whenusing the --new option.

Please define all weekdays (Mon-Fri) separately as"weekdays" value is removed.

The default entry for weekdays has beenremoved therefore, you must now defineprime and non-prime time hours for Mondaythrough Friday separately. For instance, defineprime and non-prime time hours by issuing thefollowing command for each day of the week(Monday through Friday):

pbsa-config-holidays DAY --prime-time=HH:MM --non-prime-time=HH:MM

Please define both weekends (Sat-Sun) separately as"weekends" value is removed.

The default entry for weekends has beenremoved therefore, you must now specifyprime and non-prime time hours for Saturdayand Sunday separately. For instance, define the

Proprietary Information of Altair Engineering

Page 107: Control Administrator's Guide - Altair PBS Works · Altair Control 2019.1 Administrator's Guide iv • Altair Embed SE™ ©1989-2019 (formerly solidThinking Embed® SE) • Altair

Control Administrator's Guide8 Analyze Commands p.102

Message Text Description

prime and non-prime time hours by issuing thefollowing commands:

pbsa-config-holidays Saturday --prime-time=HH:MM --non-prime-time=HH:MM

pbsa-config-holidays Sunday --prime-time=HH:MM --non-prime-time=HH:MM

Please make sure both weekend days (Sat & Sun) aredefined.

Warning message that requests that prime andnon-prime time hours be configured for bothSaturday and Sunday.

Incorrect number of arguments. Try pbsa-config-holidays --help.

Please review the usage information forthis command. You can obtain the usageinformation by using the --help option.

Config directory does not exists. Path XXXX. The configuration directory PC_EXEC/portal/services/pbsa/parser/modules/pbsdata/config does not exist. An error may haveoccurred during installation. Please re-installthe product.

Unable to instantiate the PBSA ToolKit. Please checkyour PBSWORKS_SERVER_HOME variable.

The environment variablePBSWORKS_SERVER_HOME may not havebeen exported to the environment. Contactyour systems administrator to set up theenvironment properly.

Unable to open the “holiday_<year>.conf ” file. Please check the existence and permissionsof the holiday_YYYY.conf file located atPC_EXEC/portal/services/pbsa/parser/modules/pbsdata/config

The startdate XXXX provided is not valid. Value of --start-date must be in the formatYYYY-MM-DD.

The enddate XXXX provided is not valid. Value of --end-date must be in the formatYYYY-MM-DD or now.

The startdate XXXX provided must not be greaterthan end date XXXX.

Invalid date range. Value of --start-datemust be less than the value of --end-date.

The startdate year XXXX and enddate year XXXXmust be the same.

Specified date range must not span overseveral years.

Proprietary Information of Altair Engineering

Page 108: Control Administrator's Guide - Altair PBS Works · Altair Control 2019.1 Administrator's Guide iv • Altair Embed SE™ ©1989-2019 (formerly solidThinking Embed® SE) • Altair

Control Administrator's Guide8 Analyze Commands p.103

Message Text Description

Start date format is incorrect. Expected is YYYY-MM-DD.

Dates should be in the format YYYY-MM-DD.

The primetime XXXX provided is not valid. Value of --prime-time must be in the formatof HH:MM in the 24 hour clock, all, or none.

The nonprimetime XXXX provided is not valid. Value of --non-prime-time must be in theformat of HH:MM in the 24 hour clock,all, ornone.

Proprietary Information of Altair Engineering

Page 109: Control Administrator's Guide - Altair PBS Works · Altair Control 2019.1 Administrator's Guide iv • Altair Embed SE™ ©1989-2019 (formerly solidThinking Embed® SE) • Altair

Control Administrator's Guide8 Analyze Commands p.104

8.3 Configure Custom Groups

Namepbsa-config-groups

Syntaxpbsa-config-groups [ options ] USER CUSTOMGROUP

DescriptionAdd a USER to a CUSTOMGROUP. Custom groups are used to track utilization at a project level for sitesrunning versions of PBS Professional prior to 11.2. When a user is added to a custom group, they areessentially being added to a project.

The preferred method for tracking utilization at a project level is the use of the -P option whensubmitting a job via the qub command. However, for older versions of PBS Professional, use pbsa-config-groups to define a project (custom group).

You can set the date for when a user begins working on project and when the user stops working on theproject using the options --start-date and --end-date.

By default, a custom group “OTHERS” is configured for the years 2011 through the current year.

This command must be executed as the PBSWorks Service User, root or as a user with sudo privilegesusing the sudo command on the machine where Analyze is installed.

Options--config-dir=DIRECTORY

Specify the DIRECTORY where the output of this command is directed. Defaults to PC_HOME/portal/services/pbsa/parser/modules/pbsdata/config.

--end-date=YYYY-MM-DD|nowDefine the date when the user stopped working on the project. Default is now which denotes thatthe user is still currently working in the group.

--helpDisplay help for the command. This option cannot be used in conjunction with any other options.

--newOverwrite any existing custom group configuration.

--showDisplay the current configuration on the screen for review. Use this option in conjunction withthe --start-date option to view a configuration from a previous year. Default is to display thecurrent year’s configuration.

--start-date=YYYY-MM-DDDefine the date when the user began working on the project. Default is the current date.

Proprietary Information of Altair Engineering

Page 110: Control Administrator's Guide - Altair PBS Works · Altair Control 2019.1 Administrator's Guide iv • Altair Embed SE™ ©1989-2019 (formerly solidThinking Embed® SE) • Altair

Control Administrator's Guide8 Analyze Commands p.105

--versionDisplay the command's version number. This option cannot be used in conjunction with any otheroptions.

Examples

Assign a User to a Custom Group By Designating a Start and End Date

Assign a user (tsmith) to a custom group (CFD) from January 22, 2016 to November 23, 2016.

pbsa-config-groups tsmith CFD --start-date=2016-01-22 --end-date=2016-11-23

Assign a User to a Custom Group Using the Defaults

Assign a user to a custom group with a start date of today’s date and an end date of nowdenoting that the user is currently assigned to that custom group.

pbsa-config-groups tsmith CFD

Assign a User to a Custom Group Using a Default End Date

Assign a user (tsmith) to a custom group (CFD) from January 1, 2016 with an end date of nowdenoting that the user is currently assigned to that custom group.

pbsa-config-groups tsmith CFD --start-date=2016-01-01

Assign a User to Several Custom Groups

Assign a user (tsmith) to a custom group (CFD) from January 1, 2016 to June 1, 2016, and toanother custom group (CAD) from January 1, 2016 to September 1, 2016.

pbsa-config-groups tsmith CFD --start-date=2016-01-01--end-date=2016-06-01pbsa-config-groups tsmith CAD --start-date=2016-01-01 --end-date=2016-09-01

Redirect the Output of the Command to an Alternate Location

Redirect the output of the command to an alternate location allowing the output of thecommand to be saved without overwriting the contents of the default configuration directory.

pbsa-config-groups tsmith CFD --start-date=2016-01-01 --end-date=2016-06-01 --config-dir=/tmp/pbsa-config-dir

Refresh the Custom Groups

Overwrite the currently defined custom groups. Use this command when you need to discardthe current configuration and start afresh.

pbsa-config-groups tsmith CFD --start-date=2016-01-22 --end-date=2016-11-23 --new

Show Previous Year’s Node Detailspbsa-config-nodes --start-date=2014-01-01 --show

Proprietary Information of Altair Engineering

Page 111: Control Administrator's Guide - Altair PBS Works · Altair Control 2019.1 Administrator's Guide iv • Altair Embed SE™ ©1989-2019 (formerly solidThinking Embed® SE) • Altair

Control Administrator's Guide8 Analyze Commands p.106

Error Messages

Table 6: Error Messages

Message Text Description

The PBSWORKS_COMMAND_HOME orPBSWORKS_SERVER_HOME variablewas not set, and I was unable to find oropen a pbsworks-pbsa.conf file.

The environment variables PBSWORKS_COMMAND_HOMEand PBSWORKS_SERVER_HOME have not been exported tothe environment. Contact your systems administrator to setup the environment properly.

Error occurred while settingPBSWORKS_COMMAND_HOME orPBSWORKS_SERVER_HOME

The environment variables PBSWORKS_COMMAND_HOMEand PBSWORKS_SERVER_HOME have not been exported tothe environment. Contact your systems administrator to setup the environment properly.

Python version XX is not supported.Supported version is 2.4.x or higher.

Please update the Python version to 2.4 or higher.

Error occurred while get uid for userXXXX.

Unable to get user id for current user. Contact your systemadministrator to report this error.

Operation not permitted, Please runcommand as XXXX user or ROOT.

The user does not have permissions to run this command.Please run the command as the PBSWorks Service Userspecified during installation or as root user.

The configuration file is empty. This error may occur after executing this command usingthe --show option. The configuration file is empty, thereforenothing has been configured yet.

Incorrect number of arguments. Trypbsa-config-groups --help.

Please review the usage information for this command. Youcan obtain the usage information by using the --help option.

Invalid option --remove in thecommand.

Cannot specify the --remove option when using the --newoption.

WARN: User "system" can't beremoved.

The default custom group “OTHERS” cannot be deleted.Analyze requires this entry.

Config directory does not exists. PathXXXX.

The configuration directory PC_HOME/portal/services/pbsa/parser/modules/pbsdata/config does not exist. Anerror may have occurred during installation. Please re-installthe product.

Unable to instantiate the PBSAToolKit. Please check yourPBSWORKS_SERVER_HOME variable.

The environment variable PBSWORKS_SERVER_HOME maynot have been exported to the environment. Contact yoursystems administrator to set up the environment properly.

Proprietary Information of Altair Engineering

Page 112: Control Administrator's Guide - Altair PBS Works · Altair Control 2019.1 Administrator's Guide iv • Altair Embed SE™ ©1989-2019 (formerly solidThinking Embed® SE) • Altair

Control Administrator's Guide8 Analyze Commands p.107

Message Text Description

Unable to open the“custom_group_<year>.conf ” file.

Please check the existence and permissions of thecustom_group_YYYY.conf file located at PC_HOME/portal/services/pbsa/parser/modules/pbsdata/config.

The startdate XXXX provided is notvalid.

Value of --start-date must be in the format YYYY-MM-DD.

The enddate XXXX provided is notvalid.

Value of --end-date must be in the format YYYY-MM-DD ornow.

The startdate XXXX provided must notbe greater than end date XXXX.

Invalid date range. Value of --start-date must be less thanthe value of --end-date.

The startdate year XXXX and enddateyear XXXX must be the same.

Specified date range must not span over several years.

Start date format is incorrect. Expectedis YYYY-MM-DD.

Dates should be in the format YYYY-MM-DD.

Proprietary Information of Altair Engineering

Page 113: Control Administrator's Guide - Altair PBS Works · Altair Control 2019.1 Administrator's Guide iv • Altair Embed SE™ ©1989-2019 (formerly solidThinking Embed® SE) • Altair

Control Administrator's Guide8 Analyze Commands p.108

8.4 Configure Exit Codes

Namepbsa-config-exits

Syntaxpbsa-config-exits [ options ] { EXIT_CODE [ , EXIT_CODE ] | EXIT_CODE - EXIT_CODE }

DescriptionMap an EXIT_CODE to a REASON. Use the option --reasonto specify the REASON.

By default, three exit reasons “Successful”, “Unsuccessful” and "Still Running" are mapped to specificexit codes. Use the --show option to view the defaults.

This command must be executed as the PBSWorks Service User, root or as a user with sudo privilegesusing the sudo command on the machine where Analyze is installed.

Options--application=APPLICATION

Categorize the exit codes by APPLICATION or solver. APPLICATION directly correlates to theapplication name specified via the PBS Professional “software” attribute.

Default is All applications.

--config-dir=DIRECTORYSpecify the DIRECTORY where the output of this command is directed. Defaults to PC_HOME/portal/services/pbsa/parser/modules/pbsdata/config.

--helpDisplay help for the command. This option cannot be used in conjunction with any other options.

--newOverwrite any existing exit code configuration.

--reason=REASONSpecify the exit reason. Default is Unspecified.

--removeRemove the specified exit reason/application from the current configuration. REASON defaults toUnspecified and APPLICATION defaults to All.

--showDisplay the current configuration on the screen for review.

--versionDisplay the command's version number. This option cannot be used in conjunction with any otheroptions.

Proprietary Information of Altair Engineering

Page 114: Control Administrator's Guide - Altair PBS Works · Altair Control 2019.1 Administrator's Guide iv • Altair Embed SE™ ©1989-2019 (formerly solidThinking Embed® SE) • Altair

Control Administrator's Guide8 Analyze Commands p.109

Examples

Map an Exit Code to an Exit Reason for an Application

Map an exit code of “1” to the exit reason “Successful” for the application PBSJobApp.

pbsa-config-exits 1 --reason=Successful --application=PBSJobApp

Map a List of Exit Codes to an Exit Reason for an Application

Map a list of exit codes to the exit reason “Unsuccessful” for the application PBSJobApp.

pbsa-config-exits 2,143,255 --reason=Unsuccessful --application=PBSJobApp

Map a List of Exit Codes to an Exit Reason for All Applications

Map a list of exit codes to the exit reason “Unspecified” for all applications (default).

pbsa-config-exits 2,143,255

Map a Range of Exit Codes to an Exit Reason for an Application

Map a range of exit codes to the exit reason “Unsuccessful” for the application PBSJobApp.

pbsa-config-exits 10-20 --reason=Unsuccessful --application=PBSJobApp

Tip: Use a comma separated list to map a range of negative exit codes to anexit reason.

Map Both a List and a Range of Exit Codes to an Exit Reason

Map both a comma separated list and a range of exit codes to the exit reason “Unsuccessful” forthe application PBSJobApp.

pbsa-config-exits 1,2,3,10-20 --reason=Unsuccesful --application=PBSJobApp

Map Negative Exit Codes to an Exit Reason

Use a comma separated list to map negative exit codes to an exit reason.

pbsa-config-exits 1-10,11,12-15,-13,-14,-15

Redirect the Output of the Command

Redirect the output of the command to an alternate location allowing the output of thecommand to be saved without overwriting the contents of the default configuration directory.

pbsa-config-exits 2,143,255 --config-dir=/tmp/pbsa-config-dir

Refresh the Exit Code Configuration

Overwrite the current exit code configuration. Use this command when you need to discard thecurrent exit code configuration and start afresh.

Proprietary Information of Altair Engineering

Page 115: Control Administrator's Guide - Altair PBS Works · Altair Control 2019.1 Administrator's Guide iv • Altair Embed SE™ ©1989-2019 (formerly solidThinking Embed® SE) • Altair

Control Administrator's Guide8 Analyze Commands p.110

pbsa-config-exits 1 --reason=Successful --application=PBSJobApp --new

Remove a Exit Codes using the Defaults

Remove exit codes mapped to the exit reason “Unspecified” (default) for all applications(default) .

pbsa-config-exits --remove

Remove Exit Codes for an Application and the Default Exit Reason

Remove exit codes mapped to the exit reason “Unspecified” (default) for the application“PBSJobApp”.

pbsa-config-exits --application=PBSJobApp --remove

Remove Exit Codes for All Applications and a Specific Exit Reason

Remove exit codes mapped to the exit reason “Unsuccessful” for all applications (default).

pbsa-config-exits --reason=Unsuccessful --remove

Remove Exit Codes for a Specific Application and Exit Reason

Remove exit code mapped to the exit reason “Unsuccessful” for the application “PBSJobApp”.

pbsa-config-exits --application=PBSJobApp --reason=Unsuccessful --remove

Error Messages

Table 7: Error Messages

Message Text Description

The PBSWORKS_COMMAND_HOME orPBSWORKS_SERVER_HOME variablewas not set, and I was unable to find oropen a pbsworks-pbsa.conf file.

The environment variables PBSWORKS_COMMAND_HOMEand PBSWORKS_SERVER_HOME have not been exported tothe environment. Contact your systems administrator to setup the environment properly.

Error occurred while settingPBSWORKS_COMMAND_HOME orPBSWORKS_SERVER_HOME

The environment variables PBSWORKS_COMMAND_HOMEand PBSWORKS_SERVER_HOME have not been exported tothe environment. Contact your systems administrator to setup the environment properly.

Python version XX is not supported.Supported version is 2.4.x or higher.

Please update the Python version to 2.4 or higher.

Error occurred while get uid for userXXXX.

Unable to get user id for current user. Contact your systemadministrator to report this error.

Proprietary Information of Altair Engineering

Page 116: Control Administrator's Guide - Altair PBS Works · Altair Control 2019.1 Administrator's Guide iv • Altair Embed SE™ ©1989-2019 (formerly solidThinking Embed® SE) • Altair

Control Administrator's Guide8 Analyze Commands p.111

Message Text Description

Operation not permitted, Please runcommand as XXXX user or ROOT.

The user does not have permissions to run this command.Please run the command as the PBSWorks Service Userspecified during installation or as root user.

The configuration file is empty. This error may occur after executing this command usingthe --show option. The configuration file is empty, thereforenothing has been configured yet.

WARN: User can't update privatesettings for exitreason: "Still Running"

The default entry “Still Running” cannot be deleted. Analyzerequires this entry.

Invalid option --remove in thecommand.

Cannot specify the --remove option when using the --newoption.

Incorrect number of arguments. Trypbsa-config-exits --help..

Please review the usage information for this command. Youcan obtain the usage information by using the --help option.

Config directory does not exists. The Analyze configuration directory does not exist. An errormay have occurred during installation of Analyze. Please re-install the product.

Unable to instantiate the PBSAToolKit. Please check yourPBSWORKS_SERVER_HOME variable.

The environment variable PBSWORKS_SERVER_HOME maynot have been exported to the environment. Contact yoursystems administrator to set up the environment properly.

Unable to open the “exit_status.conf ”file.

Unable to open the “exit_status.conf ” file. Please check theexistence and permissions of the exit_status.conf file locatedat PC_HOME/config/pbsa/services/pbsa/parser/modules/pbsdata/config.

Proprietary Information of Altair Engineering

Page 117: Control Administrator's Guide - Altair PBS Works · Altair Control 2019.1 Administrator's Guide iv • Altair Embed SE™ ©1989-2019 (formerly solidThinking Embed® SE) • Altair

Control Administrator's Guide8 Analyze Commands p.112

8.5 Configure the Data Collector

Namepbsa-config-dc

Syntaxpbsa-config-dc [ options ]

DescriptionConfigure the Analyze Data Collector. If the Data Collector is currently running, then it will need to berestarted after issuing this command.

The Data Collector makes the PBS Professional accounting logs and cluster node information accessibleto Analyze. It copies the accounting logs and node information from your site’s PBS Professionalaccounting log location to an intermediate holding area every 5 minutes.

You can change the time interval, however this is only recommended for advanced users of Analyze.The size of your site’s PBS Professional complex and the number and size of the accounting logs willdictate the length of time required for copying the accounting logs.

This command must be executed as root or as a user with sudo privileges using the sudo command onthe machine where the Data Collector is installed.

Options--collection-interval=MM

Define the collection interval MM in minutes for collecting the PBS Professional accounting logs(copying them to an intermediate area prior to transporting to the machine where Analyze isinstalled).

Default is 5 minutes.

--config-dir=DIRECTORYSpecify the DIRECTORY where the output of this command is directed.

--helpDisplay help for the command. This option cannot be used in conjunction with any other options.

--showDisplay the current configuration on the screen for review.

--versionDisplay the command's version number. This option cannot be used in conjunction with any otheroptions.

Proprietary Information of Altair Engineering

Page 118: Control Administrator's Guide - Altair PBS Works · Altair Control 2019.1 Administrator's Guide iv • Altair Embed SE™ ©1989-2019 (formerly solidThinking Embed® SE) • Altair

Control Administrator's Guide8 Analyze Commands p.113

Examples

Configure the Data Collector Transport and Collection Interval

configures the Data Collector to collect the PBS Professional accounting logs and cluster nodeinformation and copy them to an intermediate area every 50 minutes.

pbsa-config-dc --collection-interval=50

Redirect the Output of the Command to an Alternate Location

Redirect the output of the command to an alternate location allowing the output of thecommand to be saved without overwriting the contents of the default configuration directory.

pbsa-config-dc --collection-interval=50 --config-dir=/tmp/pbsa-config-dir

Error Messages

Table 8: Error Messages

Message Text Description

The DC_SERVER_HOME orDC_CONF_FILE variable was not set,and I was unable to find or open apbsworks-dc.conf file.

The environment variables DC_SERVER_HOME andDC_CONF_FILE have not been exported to the environment.Contact your systems administrator to set up theenvironment properly.

Error occurred while settingDC_SERVER_HOME.

The environment variable DC_SERVER_HOME has notbeen exported to the environment. Contact your systemsadministrator to set up the environment properly.

Python version XX is not supported.Supported version is 2.4.x or higher.

Please update the Python version to 2.4 or higher.

Operation not permitted, Please runcommand as XXXX user.

The user does not have permissions to run this command.Please run the command as the PBSWorks Service Userspecified during installation or as root user.

Option XXXX argument XXXX must bein minutes.

Try dc-config --help

Then specified option must be expressed in minutes.

Error occurred while get uid for userXXXX.

Unable to get user id for current user. Contact your systemadministrator to report this error.

Proprietary Information of Altair Engineering

Page 119: Control Administrator's Guide - Altair PBS Works · Altair Control 2019.1 Administrator's Guide iv • Altair Embed SE™ ©1989-2019 (formerly solidThinking Embed® SE) • Altair

Control Administrator's Guide8 Analyze Commands p.114

Message Text Description

Operation not permitted, Please runcommand as

XXXX user or ROOT.

The user does not have permissions to run this command.Please run the command as the PBSWorks Service Userspecified during installation or as root user.

Please specify --collection-interval=COLLECTION_INTERVALoption. Try dc-config --help.

You must specify a collection interval.

The configuration file is empty. This error may occur after executing this command usingthe --show option. The configuration file is empty, thereforenothing has been configured yet.

Unable to instantiate the PBSAToolKit. Please check yourPBSWORKS_SERVER_HOME variable.

The environment variable PBSWORKS_SERVER_HOME maynot have been exported to the environment. Contact yoursystems administrator to set up the environment properly.

Unable to read the dc config file. Please check the existence and permissions of the config.xmland/or modules.xml files located at PC_HOME/config/dc/config.

Proprietary Information of Altair Engineering

Page 120: Control Administrator's Guide - Altair PBS Works · Altair Control 2019.1 Administrator's Guide iv • Altair Embed SE™ ©1989-2019 (formerly solidThinking Embed® SE) • Altair

Control Administrator's Guide8 Analyze Commands p.115

8.6 Configure the Parser

Namepbsa-config-parser

Syntaxpbsa-config-parser [ options ]

DescriptionThe Parser reads the PBS Professional accounting logs, parses the appropriate information from theaccounting logs, and then loads this information into a database used by Analyze. Use the pbsa-config-parser command to configure the Parser. Analyze must be restarted after issuing thiscommand.

You can configure when the daily Parser cycle will run. Currently this cycle should not be run more thanonce in a 24-hour period. By default, the Parser cycle is configured to run once a day at 1:00 am.

This command must be executed as the PBSWorks Service User, root or as a user with sudo privilegesusing the sudo command on the machine where Analyze is installed.

Options--config-dir=DIRECTORY

Specify the DIRECTORY where the output of this command is directed.

--helpDisplay help for the command. This option cannot be used in conjunction with any other options.

--job-duration=DAYS

Define the maximum number of DAYS a running job will be considered for parsing, beyondwhich the Parser will write the PBS Professional “S” (running job) record to the unsupporteddirectory. For example, if DAYS is set to thirty, then a job that runs for more than thirty dayswill not be parsed and will be written to the unsupported directory. The unsupported directoryis where accounting logs records that fail the validation criteria. When an accounting log recordfails validation, the entry is written to file in a unsupported directory. A unsupported directoryis created for each Data Collector. The unsupported directories are located in: PC_EXEC/pbsa/services/pbsa/parser/modules/pbsdata.

--parser-timer=TIMER

Define when the daily parsing cycle should run. TIMER is a Quartz Job Scheduling Library Cronexpression specified between double quotes. For example, "0 0 1 * * ?" specifies that the Parsingcycle should run at 1:00 am every day. According to the documentation at http://www.quartz-scheduler.org, “Cron-Expressions are strings that are actually made up of seven sub-expressions,that describe individual details of the schedule. These sub-expression are separated with white-space, and represent:

• Seconds

Proprietary Information of Altair Engineering

Page 121: Control Administrator's Guide - Altair PBS Works · Altair Control 2019.1 Administrator's Guide iv • Altair Embed SE™ ©1989-2019 (formerly solidThinking Embed® SE) • Altair

Control Administrator's Guide8 Analyze Commands p.116

• Minutes

• Hours

• Day-of-Month

• Month

• Day-of-Week

• Year (optional field)”

--showDisplay the current configuration on the screen for review.

--versionDisplay the command's version number. This option cannot be used in conjunction with any otheroptions.

Examples

Configure the Parser

Configure the Parser to run the daily parsing cycle at 2:00 am.

pbsa-config-parser --parser-timer="0 0 2 * * ?" --job-duration=60

Redirect the Output of the Command to an Alternate Location

Redirect the output of the command to an alternate location allowing the output of thecommand to be saved without overwriting the contents of the default configuration directory.

pbsa-config-parser --parser-timer="0 0 2 * * ?" --config-dir=/tmp/pbsa-config-dir

Error Messages

Table 9: Error Messages

Message Text Description

The PBSWORKS_COMMAND_HOME orPBSWORKS_SERVER_HOME variable was not set, andI was unable to find or open a pbsworks-pbsa.conffile.

The environment variablesPBSWORKS_COMMAND_HOME andPBSWORKS_SERVER_HOME have not beenexported to the environment. Contactyour systems administrator to set up theenvironment properly.

Error occurred while settingPBSWORKS_COMMAND_HOME orPBSWORKS_SERVER_HOME

The environment variablesPBSWORKS_COMMAND_HOME andPBSWORKS_SERVER_HOME have not beenexported to the environment. Contactyour systems administrator to set up theenvironment properly.

Proprietary Information of Altair Engineering

Page 122: Control Administrator's Guide - Altair PBS Works · Altair Control 2019.1 Administrator's Guide iv • Altair Embed SE™ ©1989-2019 (formerly solidThinking Embed® SE) • Altair

Control Administrator's Guide8 Analyze Commands p.117

Message Text Description

Python version XX is not supported. Supportedversion is 2.4.x or higher.

Please update the Python version to 2.4 orhigher.

Error occurred while get uid for user XXXX. Unable to get user id for current user. Contactyour system administrator to report this error.

Operation not permitted, Please run command asXXXX user or ROOT.

The user does not have permissions to runthis command. Please run the command asthe PBSWorks Service User specified duringinstallation or as root user.

Option XXXX argument XXXX must be number. Trypbsa-config-parser --help

The specified option must be expressed as anumeric.

Please specify at least one option either of--parser-timer=PARSER_TIMER, or --jobduration=JOB_DURATION.

You must specify at least one of the followingoptions:

--parser-timer

--job-duration

The configuration file is empty. This error may occur after executing thiscommand using the --show option. Theconfiguration file is empty, therefore nothinghas been configured yet.

Config directory does not exists. The Analyze configuration directory does notexist. An error may have occurred duringinstallation of Analyze. Please re-install theproduct.

Unable to instantiate the PBSA ToolKit. Please checkyour PBSWORKS_SERVER_HOME variable.

The environment variablePBSWORKS_SERVER_HOME may not havebeen exported to the environment. Contactyour systems administrator to set up theenvironment properly.

Unable to read the parser config file. Parser configuration file does not exist. Contactyour system administrator.

Proprietary Information of Altair Engineering

Page 123: Control Administrator's Guide - Altair PBS Works · Altair Control 2019.1 Administrator's Guide iv • Altair Embed SE™ ©1989-2019 (formerly solidThinking Embed® SE) • Altair

Control Administrator's Guide8 Analyze Commands p.118

8.7 Reset the Analyze DatasetDiscard all Analyze data, reset the Data Collectors and reset the database password to its default value.

Namepbsa-data-reset

Syntaxpbsa-data-reset [ options ]

DescriptionUse the pbsa-data-reset command to:

• discard the data from the Analyze database.

• optionally, clear all node information from Analyze.

• delete the history files indicating which PBS Professional accounting log files have already beentransported to the machine where Analyze is installed.

• reset all Data Collectors.

• reset the Analyze database password back to “pbsworks@123”.

Note: It is recommended to use the pbsa-data-password command to change thisdefault password to a site specific one.

Stop Control before issuing this command. If this command is executed when Control is running, thenthis command will exit with an error.

Important: All simulation and model data must be removed when resetting the Analyzedataset. For more information see Delete All Simulation and Model Data.

After issuing the pbsa-data-reset command, you are given the option to delete all HPC complex nodeinformation. It is recommended to use this option when node information has become corrupted and atotal refresh of node details needs to be done for an HPC complex.

By choosing to remove the node data, the following actions take place:

1. all nodedetail_<YYYY>.conf files are removed from PC_HOME/config/pbsa/services/pbsa/parser/modules/pbsdata/config.

2. the nodedetails.xml file is removed from PC_HOME/config/pbsa/services/pbsa/parser/modules/pbsdata/config

3. Node related data is cleared from the Data Collectors.

Once the reset is complete, Data Collectors will send fresh data (i.e., PBS Professional accounting logs,scheduler logs, and node information) to Analyze. For more information see Analyze Data Forwarded bythe Data Collector.

Proprietary Information of Altair Engineering

Page 124: Control Administrator's Guide - Altair PBS Works · Altair Control 2019.1 Administrator's Guide iv • Altair Embed SE™ ©1989-2019 (formerly solidThinking Embed® SE) • Altair

Control Administrator's Guide8 Analyze Commands p.119

By choosing to NOT remove the node data, the following actions take place:

1. all nodedetail_<YYYY>.conf files are retained.

2. the nodedetails.xml file is retained.

3. Node related data is not reset on the Data Collectors.

Data Collectors will send fresh accounting and scheduler logs, but will not send a refresh of the nodeinformation. It will only send node details if there is any change in the state of the nodes.

This command must be executed as the PBSWorks Service User on the machine where Analyze isinstalled.

Options--help

Display help for the command. This option cannot be used in conjunction with any other options.

--versionDisplay the command's version number. This option cannot be used in conjunction with any otheroptions.

Error Messages

Table 10: Error Messages

Message Text Description

ThePBSWORKS_COMMAND_HOMEor PBSWORKS_SERVER_HOMEvariable was not set, and Iwas unable to find or open apbsworks-pbsa.conf file.

The environment variables PBSWORKS_COMMAND_HOME andPBSWORKS_SERVER_HOME have not been exported to theenvironment. Contact your systems administrator to set up theenvironment properly.

Error occurred while settingPBSWORKS_COMMAND_HOME orPBSWORKS_SERVER_HOME

The environment variables PBSWORKS_COMMAND_HOME andPBSWORKS_SERVER_HOME have not been exported to theenvironment. Contact your systems administrator to set up theenvironment properly.

Python version XX is notsupported. Supported version is2.4.x or higher.

Please update the Python version to 2.4 or higher.

Error occurred while get uid foruser XXXX.

Unable to get user id for current user. Contact your systemadministrator to report this error.

Operation not permitted, Pleaserun command as XXXX user orROOT.

The user does not have permissions to run this command.Please run the command as the PBSWorks Service Userspecified during installation or as root user.

Proprietary Information of Altair Engineering

Page 125: Control Administrator's Guide - Altair PBS Works · Altair Control 2019.1 Administrator's Guide iv • Altair Embed SE™ ©1989-2019 (formerly solidThinking Embed® SE) • Altair

Control Administrator's Guide8 Analyze Commands p.120

Message Text Description

Error in reading the inputs.

Please check log file: XXXX

An error occurred while retrieving system arguments. Contactyour system administrator to report this error.

Property with key XXXX doesn'texists.

An error occurred while reading the database configurationfile monetdb- pbsa.conf. Contact your system administrator toreport this error.

File XXXX must end with(.backup)

An error occurred while resetting the dataset. Contact yoursystem administrator to report this error.

File XXXX doesn't exists. An error occurred while reading the DBConnection.propertiesfile. Contact your system administrator to report this error.

Error in reading monetdb-pbsa.conf file

Please check log file: XXXX

An error occurred while reading the database configuration filemonetdb-pbsa.conf. Contact your system administrator to reportthis error.

Updation of config filedatasources.xml failed.

An error occurred while updating the configuration filedatasources.xml. Contact your system administrator to reportthis error.

Updation of config file .monetdbfailed.

An error occurred while updating the configuration

file monetdb-pbsa.conf. Contact your

system administrator to report this error.

Updation of config file monetdb-user.sql failed.

An error occurred while updating the file

monetdb-user.sql. Contact your system administrator

to report this error.

Error in splitting command: XXXX An error occurred while resetting the dataset. Contact yoursystem administrator to report this error.

Error in executing command:XXXX

An error occurred while resetting the dataset. Contact yoursystem administrator to report this error.

Command splitting failed: XXXX An error occurred while resetting the dataset. Contact yoursystem administrator to report this error.

Proprietary Information of Altair Engineering

Page 126: Control Administrator's Guide - Altair PBS Works · Altair Control 2019.1 Administrator's Guide iv • Altair Embed SE™ ©1989-2019 (formerly solidThinking Embed® SE) • Altair

Control Administrator's Guide8 Analyze Commands p.121

Message Text Description

Server startup failed. An error occurred upon database server start up. Contact yoursystem administrator to report this error.

Server status setting failed. An error occurred while setting the database server status.Contact your system administrator to report this error.

Server status check failed. An error occurred while setting the database server status.Contact your system administrator to report this error.

Datacollectors load historydeletion failed.

An error occurred while deleting the Data Collector load historyfile history.xml. Contact your system administrator to report thiserror.

Parser availability directorydeletion failed.

An error occurred while deleting the Parser availability directory.Contact your system administrator to report this error.

PBSA parser cleaning failed. An error occurred while resetting the dataset. Contact yoursystem administrator to report this error.

Data reset failed.

Please check log file: XXXX

An error occurred while resetting the dataset. Contact yoursystem administrator to report this error.

Config directory does not exists. The Analyze configuration directory does not exist. An error mayhave occurred during installation of Analyze. Please re-installthe product.

Unable to instantiate the PBSAToolKit. Please check yourPBSWORKS_SERVER_HOMEvariable.

The environment variable PBSWORKS_SERVER_HOME may nothave been exported to the environment. Contact your systemsadministrator to set up the environment properly.

Please make sure PBSA engineand Monetdb Server is notrunning.

Cannot reset the data as the Analyze service and database arerunning. Stop the Analyze service.

Proprietary Information of Altair Engineering

Page 127: Control Administrator's Guide - Altair PBS Works · Altair Control 2019.1 Administrator's Guide iv • Altair Embed SE™ ©1989-2019 (formerly solidThinking Embed® SE) • Altair

Control Administrator's Guide8 Analyze Commands p.122

8.8 Change the Analyze Database PasswordDiscard all Analyze data, reset the Data Collectors and reset the database password to its default value.

Namepbsa-data-password

Syntaxpbsa-data-password [ options ] { OLD_PASSWORD NEW_PASSWORD }

DescriptionUse the pbsa-data-password command to change or reset the Analyze database password fromOLD_PASSWORD to NEW_PASSWORD.

A database superuser is created when Analyze is installed having the following credentials: username is"pbsworks" and password is "pbsworks@123".

Stop Analyze before issuing this command. If this command is executed when the Analyze service anddatabase are running, then this command will exit with an error.

This command must be executed as the PBSWorks Service User, root or as a user with sudo privilegesusing the sudo command on the machine where Analyze is installed.

Options--help

Display help for the command. This option cannot be used in conjunction with any other options.

--versionDisplay the command's version number. This option cannot be used in conjunction with any otheroptions.

Examples

Reset the Analyze database password

Change the current password “pwdJan2018” to a new password “pwd-Feb2018”.

pbsa-data-password pwdJan2018 pwdFeb2018

Error Messages

Table 11: Error Messages

Message Text Description

The PBSWORKS_COMMAND_HOME orPBSWORKS_SERVER_HOME variable

The environment variables PBSWORKS_COMMAND_HOMEand PBSWORKS_SERVER_HOME have not been exported to

Proprietary Information of Altair Engineering

Page 128: Control Administrator's Guide - Altair PBS Works · Altair Control 2019.1 Administrator's Guide iv • Altair Embed SE™ ©1989-2019 (formerly solidThinking Embed® SE) • Altair

Control Administrator's Guide8 Analyze Commands p.123

Message Text Description

was not set, and I was unable to find oropen a pbsworks-pbsa.conf file.

the environment. Contact your systems administrator to setup the environment properly.

Error occurred while settingPBSWORKS_COMMAND_HOME orPBSWORKS_SERVER_HOME

The environment variables PBSWORKS_COMMAND_HOMEand PBSWORKS_SERVER_HOME have not been exported tothe environment. Contact your systems administrator to setup the environment properly.

Incorrect number of arguments. Trypbsa-data-password --help.

Please review the usage information for this command. Youcan obtain the usage information by using the --help option.

Python version XX is not supported.Supported version is 2.4.x or higher.

Please update the Python version to 2.4 or higher.

Error occurred while get uid for userXXXX.

Unable to get user id for current user. Contact your systemadministrator to report this error.

Operation not permitted, Please runcommand as XXXX user or ROOT.

The user does not have permissions to run this command.Please run the command as the PBSWorks Service Userspecified during installation or as root user.

Error in reading the inputs. Pleasecheck log file: XXXX

An error occurred while retrieving system arguments.Contact your system administrator to report this error.

Property with key XXXX doesn't exists. An error occurred while reading the database configurationfile monetdb- pbsa.conf. Contact your system administratorto report this error.

File XXXX must end with (.backup) An error occurred while resetting the dataset. Contact yoursystem administrator to report this error.

File XXXX doesn't exists. An error occurred while reading the DBConnection.propertiesfile. Contact your system administrator to report this error.

Error in reading monetdb-pbsa.conf file

Please check log file: XXXX

An error occurred while reading the database configurationfile monetdb-pbsa.conf. Contact your system administrator toreport this error.

Error in splitting command: XXXX An error occurred while resetting the dataset. Contact yoursystem administrator to report this error.

Proprietary Information of Altair Engineering

Page 129: Control Administrator's Guide - Altair PBS Works · Altair Control 2019.1 Administrator's Guide iv • Altair Embed SE™ ©1989-2019 (formerly solidThinking Embed® SE) • Altair

Control Administrator's Guide8 Analyze Commands p.124

Message Text Description

Error in executing command: XXXX An error occurred while resetting the dataset. Contact yoursystem administrator to report this error.

Line must be (key=value). xxxxxxx There is a problem with the DBConnection.properties file.Contact your system administrator to report this error.

Updation of config file context.xmlfailed.

An error occurred while updating the configuration filecontext.xml. Contact your system administrator to report thiserror.

Updation of config file .monetdb failed. An error occurred while updating the configuration filemonetdb-pbsa.conf. Contact your system administrator toreport this error.

Updation of config file monetdb-user.sqlfailed.

An error occurred while updating the file monetdb-user.sql.Contact your system administrator to report this error.

Command splitting failed: XXXX An error occurred while resetting the dataset. Contact yoursystem administrator to report this error.

Server startup failed. An error occurred upon database server start up. Contactyour system administrator to report this error.

Server status setting failed. An error occurred while setting the database server status.Contact your system administrator to report this error.

Server status check failed. An error occurred while setting the database server status.Contact your system administrator to report this error.

Password change failed. Please checklog file: XXXX

An error occurred while changing the password. Contact yoursystem administrator to report this error.

Please make sure PBSA engine andMonetdb Server is not running.

Cannot reset the data as the Analyze service and databaseare running. Stop the Analyze service.

Proprietary Information of Altair Engineering

Page 130: Control Administrator's Guide - Altair PBS Works · Altair Control 2019.1 Administrator's Guide iv • Altair Embed SE™ ©1989-2019 (formerly solidThinking Embed® SE) • Altair

Control Administrator's Guide8 Analyze Commands p.125

8.9 Create a Diagnosis File for TroubleshootingIssuesCapture Analyze information that could assist with diagnosing issues.

Namepbsa-setup-diagnosis

Syntaxpbsa-setup-diagnosis [ options ]

DescriptionThis command creates a zip file in the current working directory containing information that can assistin the resolution of system issues. This file can be emailed to Altair Systems Support to help with thediagnosis of an issue.

This command must be executed as the PBSWorks Service User, root or as a user with sudo privilegesusing the sudo command on the machine where Analyze is installed.

Options--help

Display help for the command. This option cannot be used in conjunction with any other options.

--versionDisplay the command's version number. This option cannot be used in conjunction with any otheroptions.

Error Messages

Table 12: Error Messages

Message Text Description

Error occurred while findingconfiguration Path <FILENAME>

The Analyze configuration file denoted by <FILENAME>cannot be found. Contact your systems administrator.

Error while creating temporarydirectory

An error occurred while creating the zip file. Contact yoursystems administrator.

Error while read source <DIRECTORY>.Please make sure directory exist anduser have proper permission

An error occurred while creating the zip file. Contact yoursystems administrator.

Proprietary Information of Altair Engineering

Page 131: Control Administrator's Guide - Altair PBS Works · Altair Control 2019.1 Administrator's Guide iv • Altair Embed SE™ ©1989-2019 (formerly solidThinking Embed® SE) • Altair

Cloud Bursting 9

9 Cloud Bursting

This chapter covers the following:

• 9.1 Cloud Bursting Overview (p. 127)

• 9.2 Cloud Bursting Terminology (p. 129)

• 9.3 Prerequisites for Cloud Bursting (p. 130)

• 9.4 Cloud Bursting Startup Script (p. 131)

• 9.5 Configure Amazon Web Service Cloud Bursting (p. 134)

• 9.6 Configuring Microsoft Azure Cloud Bursting (p. 145)

• 9.7 Configure Google Cloud Platform Cloud Bursting (p. 157)

• 9.8 Configure Oracle Cloud Platform Cloud Bursting (p. 165)

• 9.9 Configure Orange Cloud Flexible Engine for Cloud Bursting (p. 180)

• 9.10 Configure HUAWEI Cloud for Cloud Bursting (p. 191)

• 9.11 Configure Open Telekom Cloud for Cloud Bursting (p. 201)

• 9.12 Add Quotas and Alerts (p. 211)

• 9.13 Add a Token (p. 212)

• 9.14 Configure PBS Professional for Cloud Bursting (p. 213)

• 9.15 Troubleshoot Cloud Bursting (p. 228)

• 9.16 Manage Cloud and On-Premise Jobs (p. 229)

• 9.17 Run Cloud Jobs On-Premise Before Bursting (p. 231)

• 9.18 Override Instance Type or Image at Job Submission (p. 233)

• 9.19 Request InfiniBand Nodes (p. 234)

• 9.20 AWS Spot Pricing (p. 235)

• 9.21 Modify the Bursting Hook (p. 241)

• 9.22 Disable Cloud Bursting (p. 245)

• 9.23 Uninstall PCLM CLI and Cloud Bursting Hook (p. 246)

Page 132: Control Administrator's Guide - Altair PBS Works · Altair Control 2019.1 Administrator's Guide iv • Altair Embed SE™ ©1989-2019 (formerly solidThinking Embed® SE) • Altair

Control Administrator's Guide9 Cloud Bursting p.127

9.1 Cloud Bursting OverviewA description of cloud bursting, a typical bursting setup and an overview of steps for setting up cloudbursting.

Cloud bursting is a configuration that is set up between an HPC cluster and a public cloud to deal withpeaks in cluster demand. When resource capacity at the HPC reaches a certain point, the demand isdirected to a public cloud so there is no interruption of services.

Cloud bursting requires that you already have an account with one of the supported cloud providers:

• Microsoft Azure

• Amazon Web Services (AWS)

• Google Cloud Platform (GCP)

• Oracle Cloud Platform

• Orange Cloud Flexible Engine

• Open Telekom Cloud (OTC)

• HUAWEI Cloud

Cloud bursting is managed through PBS Professional by submitting jobs to one or more designatedcloud queues. A server periodic hook monitors cloud queues, estimates the demand for cloud nodes,and dynamically adjusts the number of nodes by bursting or unbursting as needed.

Figure 21: Typical Cloud Bursting Setup

Proprietary Information of Altair Engineering

Page 133: Control Administrator's Guide - Altair PBS Works · Altair Control 2019.1 Administrator's Guide iv • Altair Embed SE™ ©1989-2019 (formerly solidThinking Embed® SE) • Altair

Control Administrator's Guide9 Cloud Bursting p.128

Steps for Cloud BurstingThere is some setup required at the cloud provider level before a cloud bursting scenario can be createdusing Control. There is also PBS Professional configuration that must be done. General steps that arerequired for bursting:

1. Create a cloud-init script that runs after a cloud node is burst.

2. Create necessary cloud provider components such as a virtual network and a virtual machine.

3. SSH into the cloud provider virtual machine and configure the VM for your site's environment suchas installing PBS, installing cloud-init, mounting file systems, connecting it to the authenticationservice, installing any applications, etc.

4. Create an image of the virtual machine.

5. Provide the necessary cloud provider details, including the cloud-init script, by creating a burstingscenario with Control.

6. Configure PBS cloud bursting custom resources.

7. Create PBS cloud queues.

8. Install the PCLM CLI and cloud bursting hook.

9. Configure the PBS cloud bursting hook.

Note: Steps vary depending upon the cloud provider used for cloud bursting.

Proprietary Information of Altair Engineering

Page 134: Control Administrator's Guide - Altair PBS Works · Altair Control 2019.1 Administrator's Guide iv • Altair Embed SE™ ©1989-2019 (formerly solidThinking Embed® SE) • Altair

Control Administrator's Guide9 Cloud Bursting p.129

9.2 Cloud Bursting TerminologyDefinitions for terminology that is used to describe cloud bursting.

BurstThe action of adding a node in the cloud, and adding it to the PBS complex.

UnburstThe action of removing a node from both the PBS complex and the cloud.

Bursting scenarioScenario created in Control containing information about resources provided by cloud service thatare used for bursting.

ImageA pre-configured OS image in the cloud from which virtual machines can be instantiated.

InstanceA single virtual machine in a cloud computing environment.

Instance typeInstance characteristics: the combination of CPU, memory, storage, and networking capacity.

api_keyScenario token generated for the bursting scenario. Identifies cloud service provider and controlsaccess to the Control cloud bursting APIs.

Proprietary Information of Altair Engineering

Page 135: Control Administrator's Guide - Altair PBS Works · Altair Control 2019.1 Administrator's Guide iv • Altair Embed SE™ ©1989-2019 (formerly solidThinking Embed® SE) • Altair

Control Administrator's Guide9 Cloud Bursting p.130

9.3 Prerequisites for Cloud BurstingA cloud provider account, licenses, an on-premise PBS Complex and a secure network connection mustbe in place before cloud bursting can be enabled.

Note: Cloud bursting is supported on Linux platforms only.

• An installation of Control 2019.1.

• An account with one of the supported cloud providers.

• PBS Works Burst Node licenses.

• An on-premise PBS complex running PBS Professional 19.2.1; this includes the PBS Server andexecution hosts. All the setup required for bursting is done on your on-premise PBS complex (it isnot done on the PBS Server installed with Control. The PBS Server installed with Control is only forrunning simulations).

• A dedicated network line, VPN setup, or SSH tunnel to the cloud provider for seamlesscommunication between the two.

Note: A DNS server is no longer a requirement for cloud bursting.

Proprietary Information of Altair Engineering

Page 136: Control Administrator's Guide - Altair PBS Works · Altair Control 2019.1 Administrator's Guide iv • Altair Embed SE™ ©1989-2019 (formerly solidThinking Embed® SE) • Altair

Control Administrator's Guide9 Cloud Bursting p.131

9.4 Cloud Bursting Startup ScriptCreate a script that is ran when the cloud node is burst.

IntroductionYour site will want to do some configuration to their cloud nodes after booting. For example, you maywant to install some packages, add users, or start services. A utility specifically designed for this iscloud-init. The cloud-init program is a bootstrapping utility for pre-provisioned disk images that run invirtualized environments, usually cloud-oriented services. Basically, it sets up the server instance to beusable when it’s finished booting. You must install cloud-init on your cloud provider VM to simplify thetask of configuring your instances on boot. For more information see cloud-init.

Several input types are supported by cloud-init.

• Shell scripts

• Cloud config files

The simplest way to configure an instance on boot is to use a shell script. This shell script is uploadedwhen a cloud bursting scenario is created. The shell script must begin with #! in order for cloud-init torecognize it as a shell script.

Below are examples of configuration that should be done via this script after a node has been burstin the cloud. These examples are not intended to be copied and pasted as is, you must configure thestartup script per your site's needs.

Example of a cloud-init Script#!/bin/sh# Map IP address to hostnames via /etc/hostsecho "/etc/hosts setup"rm -f /etc/hostsecho "127.0.0.1 localhost localhost.localdomain localhost4 localhost4.localdomain4" > /etc/hostsecho "PBS_SERVER_IP_ADDR headnode headnode.DOMAINNAME" >> /etc/hosts

# Disable NetworkManager so that it does not overwrite the /etc/resolv.conf filesystemctl disable NetworkManagersystemctl stop NetworkManagersystemctl enable networksystemctl start network

# Configure PBS via /etc/pbs.confecho "pbs setup"systemctl stop pbsrm -f /etc/pbs.confecho "PBS_EXEC=/opt/pbs/default" > /etc/pbs.confecho "PBS_HOME=/var/spool/PBS" >> /etc/pbs.confecho "PBS_START_SERVER=0" >> /etc/pbs.confecho "PBS_START_MOM=1" >> /etc/pbs.confecho "PBS_START_SCHED=0" >> /etc/pbs.confecho "PBS_START_COMM=0" >> /etc/pbs.confecho "PBS_SERVER=PBS_SERVER_HOSTNAME" >> /etc/pbs.confecho "PBS_CORE_LIMIT=unlimited" >> /etc/pbs.confecho "PBS_SCP=/bin/scp" >> /etc/pbs.confecho "PBS_LEAF_ROUTERS=HOSTNAME,HOSTNAME" >> /etc/pbs.conf

Proprietary Information of Altair Engineering

Page 137: Control Administrator's Guide - Altair PBS Works · Altair Control 2019.1 Administrator's Guide iv • Altair Embed SE™ ©1989-2019 (formerly solidThinking Embed® SE) • Altair

Control Administrator's Guide9 Cloud Bursting p.132

# Configure the MoMecho "mom config setup". /etc/pbs.confecho "\$clienthost $PBS_SERVER" >> /var/spool/pbs/mom_priv/configecho "\$clienthost ${PBS_SERVER//.*}" >> /var/spool/pbs/mom_priv/configecho "\$restrict_user_maxsysid 999" >> /var/spool/pbs/mom_priv/config

# Restart pbssystemctl start pbs

An explanation for each section of the cloud-init script is given below. For the below examples assumethe following: Fully qualified domain name (FQDN) of the PBS Server = pbs.altair.com NIC address ofthe PBS Server is 10.0.0.5 on the 10.0.0.0/24 network.

Configure the Host FileMap hostnames to the PBS Server IP address by updating the /etc/hosts file.

# Map IP address to hostnames via /etc/hostsecho "/etc/hosts setup"rm -f /etc/hostsecho "127.0.0.1 localhost localhost.localdomain localhost4 localhost4.localdomain4" > /etc/hostsecho "10.0.0.5 headnode headnode.pbs.altair.com" >> /etc/hosts

Disable NetworkManager and Use Network InterfaceStop NetworkManager and use network interface so that the contents of /etc/resolv.conf do not getoverwritten:# Disable NetworkManager so that it does not overwrite the /etc/resolv.conf filesystemctl disable NetworkManagersystemctl stop NetworkManagersystemctl enable networksystemctl start network

Configure PBSUpdate the PBS configuration file /etc/pbs.conf:# Configure pbs.confecho "pbs setup"systemctl stop pbsrm -f /etc/pbs.confecho "PBS_EXEC=/opt/pbs" > /etc/pbs.confecho "PBS_HOME=/var/spool/pbs" >> /etc/pbs.confecho "PBS_START_SERVER=0" >> /etc/pbs.confecho "PBS_START_MOM=1" >> /etc/pbs.confecho "PBS_START_SCHED=0" >> /etc/pbs.confecho "PBS_START_COMM=0" >> /etc/pbs.confecho "PBS_SERVER=PBS_SERVER_HOSTNAME" >> /etc/pbs.confecho "PBS_CORE_LIMIT=unlimited" >> /etc/pbs.confecho "PBS_SCP=/bin/scp" >> /etc/pbs.confecho "PBS_LEAF_ROUTERS=HOSTNAME,HOSTNAME" >> /etc/pbs.conf

Where PBS_SERVER_HOSTNAME represents the hostname of the machine where the PBS Server isinstalled and HOSTNAME tells each endpoint which communication daemon it should talk to.

Note: If PBS is installed in non-default home and execution directory, then update thevalue of PBS_EXEC and PBS_HOME.

Proprietary Information of Altair Engineering

Page 138: Control Administrator's Guide - Altair PBS Works · Altair Control 2019.1 Administrator's Guide iv • Altair Embed SE™ ©1989-2019 (formerly solidThinking Embed® SE) • Altair

Control Administrator's Guide9 Cloud Bursting p.133

Configure the PBS MoM and Restart PBSUpdate the PBS_HOME/mom_priv_config file to configure the MoM:

# Configure /var/spool/pbs/mom_priv/configecho "mom config setup". /etc/pbs.confecho "\$clienthost $PBS_SERVER" >> /var/spool/pbs/mom_priv/configecho "\$clienthost ${PBS_SERVER//.*}" >> /var/spool/pbs/mom_priv/configecho "\$restrict_user_maxsysid 999" >> /var/spool/pbs/mom_priv/config

systemctl start pbs

Note: If PBS is installed in non-default directory, then change the path to the PBS homedirectory (i.e., /var/spool/pbs).

Optional ConfigurationUse the startup script to configure filesystems (/etc/fstab), configure NIS (/etc/yp.conf), mountnecessary filesystems, and any other configuration that your site requires.

Proprietary Information of Altair Engineering

Page 139: Control Administrator's Guide - Altair PBS Works · Altair Control 2019.1 Administrator's Guide iv • Altair Embed SE™ ©1989-2019 (formerly solidThinking Embed® SE) • Altair

Control Administrator's Guide9 Cloud Bursting p.134

9.5 Configure Amazon Web Service Cloud BurstingCreate the necessary components in AWS that are required for cloud bursting and then create a Controlcloud bursting scenario.

Before you can begin creating the AWS components necessary for cloud bursting, you must create andactivate an AWS account. For more information see How do I create and activate a new Amazon WebServices account.

9.5.1 Create an Amazon Web Service User AccountCreate an Amazon Web Service (AWS) user account so that you can later create a cloud account inControl.

You must already have signed up for an AWS account.

AWS documentation can be found at Creating an IAM User in Your AWS Account.

Create an AWS user account and give this account administrative permissions. Once this task iscomplete, you should have downloaded a CSV file containing the following:

• Access key ID

• Secret access key

This information is necessary to create a cloud account in Control.

1. Login in the AWS console.

2. Using the search box located under AWS services, enter IAM.

3. Click the IAM search result.The Identity and Access Management dashboard is opened.

4. In the navigation pane on the left-hand side of the web page, click Users.

5. Click Add user.This user will have administrative rights to any machines hosted in the cloud.

6. Enter the following information to add a user:

a) For User name, enter a name for the user.

The name can be anything meaningful to your organization, e.g., pc_clouduser.

b) For Access type, enable Programmatic access.

The user requires this type of access because Control needs to make API calls or use theAWS CLI. An access key ID and a secret access key are generated for the user.

7. Click Next:Permissions.

8. Optional: Click Add user to group.

This button may already be selected.

9. Click Create group.

10. Enter the following information to create a group, add the user to the group, and choose apermission policy for the group.

a) For Group name, enter a group name.

Proprietary Information of Altair Engineering

Page 140: Control Administrator's Guide - Altair PBS Works · Altair Control 2019.1 Administrator's Guide iv • Altair Embed SE™ ©1989-2019 (formerly solidThinking Embed® SE) • Altair

Control Administrator's Guide9 Cloud Bursting p.135

The name can be anything meaningful to your organization, e.g., pc_cloudgroup.

b) For Policy type, enable AdministratorAccess.

This policy provides full access to AWS services and resources.

11. Click Create group.You are returned to the Add user page and the new group is enabled indicating that the user isadded to the new group.

12. Click Next: Tags.

13. Click Next: Review.

14. Click Create user.15. Click Download.csv.

16. Download and save this file in a secure location.

This file contains an access key ID and a secret access key and is used later to add a cloudaccount to Control.

17. Click Close.The new user account is displayed.

9.5.2 Add an Amazon Web Service Cloud AccountAdd an existing AWS user account to Control so that you can create a cloud bursting scenario or anappliance.

You must already created an AWS user account, given it administrative permissions and downloadeda CSV file containing an access key ID and a secret access key. The information in the CSV file isnecessary to create a cloud account in Control. If you do not have this information see Create anAmazon Web Service User Account.

1. Login to Control.

2. Click the Cloud tab.

3. Under Infrastructure, click Cloud.

4. Click .

5. Enter the following to add a cloud account:

a) For Account name, enter any name for the cloud account.

The name can be anything meaningful to your organization, e.g., aws_cloudaccount.

b) For Access Key ID, copy and paste the value of Access key ID from the downloaded CSVfile.

c) For Secret Access Key, copy and paste the Secret access key value from the downloadedCSV file.

6. Click Create account.

Proprietary Information of Altair Engineering

Page 141: Control Administrator's Guide - Altair PBS Works · Altair Control 2019.1 Administrator's Guide iv • Altair Embed SE™ ©1989-2019 (formerly solidThinking Embed® SE) • Altair

Control Administrator's Guide9 Cloud Bursting p.136

9.5.3 Create a Virtual Private Cloud NetworkCreate a Virtual Private Cloud (VPC) in AWS so that you can later create a virtual machine for cloudbursting.

Login to your AWS Management Console and choose a region based on the geographical location ofyour users. All cloud resources that are created are placed in this region. For more information seeRegions and Availability Zones. The menu for selecting a region is located at the upper right-handcorner of the AWS Console menu bar.

AWS documentation can be found at Getting Started with IPv4 for Amazon VPC and Working with VPCsand Subnets.

Figure 22: AWS Region

1. Click located in the upper left-hand corner of the web page.

2. Using the search box located under AWS services, enter VPC.

3. Click the VPC search result.The VPC dashboard is opened.

4. In the menu located on the left-hand side of the web page, click Your VPCs.

Create a virtual private cloud.

5. Click Create VPC.

6. Enter the following to create a VPC:

a) For Name, enter any name for the VPC.

The name can be anything meaningful to your organization, e.g., bursting_vpc.

b) For IPv4 CIDR block, provide an address range in CIDR notation.

c) For IPv6 CIDR block, enable No IPv6 CIDR Block.

d) For Tenancy, choose Default.7. Click Yes, Create.

Create a subnet for the VPC:

8. In the menu located on the left-hand side of the web page, click Subnets.

9. Click Create Subnet.a) For Name tag, enter a name for the subnet.

The name can be anything meaningful to your organization, e.g., bursting_subnet.

b) For VPC, choose the VPC that was previously created (e.g. bursting_vpc).

c) For Availability Zone, choose one of the following options:

• Choose an availability zone from the menu.

• Choose No Preference to let Amazon choose an Availability Zone for you.

d) For IPv4 CIDR block, provide an address range in CIDR notation.

10. Click Create.

Proprietary Information of Altair Engineering

Page 142: Control Administrator's Guide - Altair PBS Works · Altair Control 2019.1 Administrator's Guide iv • Altair Embed SE™ ©1989-2019 (formerly solidThinking Embed® SE) • Altair

Control Administrator's Guide9 Cloud Bursting p.137

11. Click Close.

9.5.4 Create an Internet GatewayCreate an internet gateway and attach it to the bursting VPC.

AWS documentation can be found at Internet Gateways.

An internet gateway must be created so that you can later SSH into the virtual machine that is used forcloud bursting.

1. Login in the AWS console.

2. Click located in the upper left-hand corner of the web page.

3. Using the search box located under AWS services, enter VPC.

4. Click the VPC search result.The VPC dashboard is opened.

5. In the menu located on the left-hand side of the web page, click Internet Gateways.

6. Click Create internet gateway.

7. Enter the following to create an internet gateway:

a) For Name tag, enter a name for the internet gateway.

The name can be anything meaningful to your organization, e.g., bursting_gateway

8. Click Create.

9. Click Close.

10. Select the internet gateway that was just created by enabling the check box next to the name ofthe gateway.

Tip: You may need to deselect any other internet gateways that are displayed in thelist. Amazon creates default resources for your selected region so a default internetgateway may already exist.

11. Click Actions > Attach to VPC.

12. Select the VPC that was created previously (e.g. bursting_vpc).

13. Click Attach.

9.5.5 Update the VPC Route TableAdd a rule to the VPC route table that allows all internet access and associate the route table to thebursting subnet.

AWS documentation can be found at Route Tables.

1. Login in the AWS console.

2. Click located in the upper left-hand corner of the web page.

Proprietary Information of Altair Engineering

Page 143: Control Administrator's Guide - Altair PBS Works · Altair Control 2019.1 Administrator's Guide iv • Altair Embed SE™ ©1989-2019 (formerly solidThinking Embed® SE) • Altair

Control Administrator's Guide9 Cloud Bursting p.138

3. Using the search box located under AWS services, enter VPC.

4. Click the VPC search result.The VPC dashboard is opened.

5. In the menu located on the left-hand side of the web page, click Route Tables.A list of route tables is displayed.

6. Select the route table attached to your VPC (e.g. bursting_vpc) by enabling the check box next tothe name of the route table.

The VPC column in the route table list specifies the VPC to which the route table is attached.

7. Click the Routes tab at the bottom of the web page.

8. Click Edit routes.

Figure 23: Add a Route

9. Click Add route.

Add a rule that allows all traffic access to the internet gateway:

10. Enter the following to add a rule:

a) For Destination enter 0.0.0.0/0.

b) For Target, select Internet Gateway and then the internet gateway that was createdpreviously (e.g. bursting_gateway).

11. Click Save routes.

12. Click Close.

Associated the route table to the bursting subnet:

13. Click the Subnet Associations tab.

14. Click Edit subnet associations.

15. Select the subnet created for cloud bursting from the list.

16. Click Save.

9.5.6 Create a Virtual MachineCreate a CentOS 7 virtual machine in AWS Elastic Compute Cloud (EC2).

AWS documentation can be found at Launch a Linux Virtual Machine and Launching a Virtual Machinewith Amazon EC2.

Proprietary Information of Altair Engineering

Page 144: Control Administrator's Guide - Altair PBS Works · Altair Control 2019.1 Administrator's Guide iv • Altair Embed SE™ ©1989-2019 (formerly solidThinking Embed® SE) • Altair

Control Administrator's Guide9 Cloud Bursting p.139

1. Login in the AWS console.

2. Click located in the upper left-hand corner of the web page.

3. Using the search box located under AWS services, enter EC2.

4. Click the EC2 search result.The EC2 dashboard is opened.

5. In the menu located on the left-hand side of the web page, click Instances.

6. Click Launch Instance.

7. In the menu located on the left-hand side of the web page, click AWS Marketplace.

8. Using the Search AWS Marketplace Products search box, enter CentOS 7.

9. Locate the CentOS 7 Amazon Machine Image (AMI) and click Select.10. Click Continue.

11. Select an Instance Type appropriate for your site's workload based on:

• the number of cores

• the amount of memory

• storage

• network performance

Consider the nature of the applications that you plan to deploy on the instance, the number ofusers that you expect to use the applications, and also how you expect the load to scale in thefuture. Remember to also factor in the CPU and memory resources that are necessary for theoperating system.

12. Click Next: Configure Instance Details.

13. Enter the following to configure instance details:

a) For Number of instances, specify 1.

b) For Network, choose the VPC that you previously created (e.g. bursting_vpc).The bursting subnet is automatically populated.

c) For Auto-assign Public IP, select Enable.

14. Click Next: Add Storage.

15. Specify the storage options as per your site's requirements.

Tip: It is recommended to enable Delete on Termination to delete EBS volumeswhen the virtual machine is terminated.

16. Click Next: Add Tags.

17. Optional: You can add tags in key-value pairs for the VM per your site's requirements.

18. Click Next: Configure Security Group.

19. Enter the following to create a security group for the VM:

a) For Assign a security group, enable Create a new security group.

b) For Security group name, enter a name for the security group.

The name can be anything meaningful to your organization, e.g., bursting_security_group.

a) Add new security rules based on your site's requirements.

Proprietary Information of Altair Engineering

Page 145: Control Administrator's Guide - Altair PBS Works · Altair Control 2019.1 Administrator's Guide iv • Altair Embed SE™ ©1989-2019 (formerly solidThinking Embed® SE) • Altair

Control Administrator's Guide9 Cloud Bursting p.140

Figure 24: Security Rules

20. Click Review and Launch.

21. Review the VM information of the VM and click Launch.

22. Create a new public/private key pair for the VM by entering the following:

a) Select Create a new key pair.b) Provide a name for the key pair.

c) Click Download Key Pair.d) Download and save this file in a secure location.

The information in this .pem file is used later to SSH into the VM.

23. Click Launch Instances.

24. At the bottom of the web page, click View Instances.All virtual machines that have been created are displayed.

Your virtual machine is ready when the Instance State is equal to "running" and Status Checks arecomplete. The virtual machine can be accessed using its IPv4 Public IP.

Figure 25: Bursting Virtual Machine

9.5.7 Installing and Configuring the PBS MoM on the VMInstall the PBS Professional MoM on the virtual machine and configure the VM per your site'senvironment.

The username for logging into the virtual machine is dependent on the Amazon Machine Image (AMI)that you used to create the virtual machine. For example, the user name for a CentOS AMI is "centos".For information about user names and instructions for connecting and copying files to the virtualmachine see Connecting to Your Linux Instance Using SSH.

You will need the .pem file downloaded while creating the VM.

A cloud-init script is provided during the creation of the bursting scenario. This script is ran after acloud node is burst and is used to set up the system environment so that nodes burst in the cloud cancommunicate with the PBS Professional Server, therefore installation of cloud-init on the VM is required.For more information about installing cloud-init, refer to the Cloudinit Documentation.

1. Login to your site's PBS Professional Server.

Proprietary Information of Altair Engineering

Page 146: Control Administrator's Guide - Altair PBS Works · Altair Control 2019.1 Administrator's Guide iv • Altair Embed SE™ ©1989-2019 (formerly solidThinking Embed® SE) • Altair

Control Administrator's Guide9 Cloud Bursting p.141

2. SSH into the virtual machine as the user "centos" using the .pem file and the IPv4 Public IPassigned to the VM:ssh -i /path/my-key-pair.pem centos@IPV4PublicIP

where /path/my-key-pair.pem is the path to the .pem file downloaded while creating the virtualmachine and IPV4PublicIP is the public IP address of the virtual machine.

Figure 26: Bursting Virtual Machine

3. Enter the command:sudo -i

4. Using the PBS Professional Installation and Upgrade Guide, install and configure the PBSProfessional MOM.

5. Once the PBS Professional MOM is installed, edit the /etc/pbs.conf file.

6. Change the value of PBS_SERVER to the hostname of the PBS Professional Server.

7. Save the file.

8. Edit the file PBS_HOME/mom_priv/config.

where PBS_HOME is the home directory of PBS Professional.

9. Change the value of $clienthost to the hostname of the PBS Professional Server.

10. Save the file.

11. Configure the VM for your site's environment such as pointing it the DNS server, mounting filesystems, connecting it to the authentication service, installing any applications, etc.

12. Install cloud-init.

13. Connect the VM to your PBS Professional server and submit a job to ensure it is working asexpected.

9.5.8 Create an OS ImageCreate an image of the virtual machine configured per your site's requirements.

You must have already create a virtual machine, installed the PBS Professional MoM on the VM andconfigured the VM per your site's environment.

AWS documentation can be found at Create an AMI from an Amazon EC2 Instance.

1. Login in the AWS console.

2. Click located in the upper left-hand corner of the web page.

3. Using the search box located under AWS services, enter EC2.

4. Click the EC2 search result.The EC2 dashboard is opened.

Proprietary Information of Altair Engineering

Page 147: Control Administrator's Guide - Altair PBS Works · Altair Control 2019.1 Administrator's Guide iv • Altair Embed SE™ ©1989-2019 (formerly solidThinking Embed® SE) • Altair

Control Administrator's Guide9 Cloud Bursting p.142

5. In the menu located on the left-hand side of the web page, click Instances.

6. Select the virtual machine created for cloud bursting by enabling the check box next to its name.

7. At the top, click Actions > Instance State > Stop.

8. Click Yes, Stop.

It may take some time for the virtual machine to be stopped.

Important: Do not proceed until the Instance State is "stopped".

9. Click Actions > Image > Create Image.

10. For Image name, enter a name for the image.

The name can be anything meaningful to your organization, e.g., bursting_image.

11. Click Create Image.

12. Click the View pending image ami-xxxxxxxxxx link.

The image is complete when its Status is "available".

You can now delete the virtual machine so that you are no longer charged for it.

9.5.9 Create an AWS Cloud Bursting ScenarioCreate an AWS bursting scenario so that PBS Professional execution nodes can be burst in the cloud.

Open a browser window and login to your AWS Management Console. Open a second browser windowand login to Control.

A cloud-init script is provided during the creation of the bursting scenario. This script is ran after acloud node is burst and is used to set up the system environment so that nodes burst in the cloud cancommunicate with the PBS Professional Server. For more information about writing cloud-init scripts,refer to the Cloudinit Documentation and the Cloud Bursting chapter of the ControlAdministrator'sGuide.

1. Navigate to the Control browser window.

2. Click Cloud.

3. Under Infrastructure, click Bursting.

4. Click Add Bursting Scenario.

5. Enter information about the bursting scenario such as name, cloud provider, cloud account, andgeographic location.

a) For Name, enter a name for the cloud bursting scenario.

b) For Description, enter a description.

c) For Cloud Account, choose a registered AWS account from the list.

d) Select a region from the list that matches the region selected in AWS.

Proprietary Information of Altair Engineering

Page 148: Control Administrator's Guide - Altair PBS Works · Altair Control 2019.1 Administrator's Guide iv • Altair Embed SE™ ©1989-2019 (formerly solidThinking Embed® SE) • Altair

Control Administrator's Guide9 Cloud Bursting p.143

Figure 27: AWS Region

6. Click Next.7. For Domain name, provide the domain name of your organization for the cloud bursting network.

altair.com, burst.altair.com

8. For Hostname prefix, enter a prefix for the names of the burst nodes.If "node" is entered , the cloud bursting nodes will be available to PBS Professional as node1,node2, node3, etc.

9. Navigate to the AWS browser window.

10. Click located in the upper left-hand corner of the web page.

11. Using the search box located under AWS services, enter EC2.

12. Click the EC2 search result.The EC2 dashboard is opened.

13. In the menu located on the left-hand side of the web page, click AMIs.

14. Select the Amazon Machine Image (AMI) created for cloud bursting by enabling the check boxnext to its name.

15. In the Details tab located at the bottom of the web page, hover over the AMI ID.

A copy to clipboard icon is displayed.

16. Click .

17. Navigate to the Control browser window.

18. For the AMI ID, paste the AMI ID copied from AWS.

19. Navigate to the AWS browser window.

20. In the menu located on the left-hand side of the web page, click Security Groups underNETWORK & SECURITY.

21. Select the Security Group created for the bursting virtual machine by enabling the check box nextto its Group ID.

22. In the Description tab located at the bottom of the web page, hover over the Group ID.

A copy to clipboard icon is displayed.

23. Click .

24. Navigate to the Control browser window.

25. For the Security group ID, paste the Group ID copied from AWS.

26. Navigate to the AWS browser window.

27. Click located in the upper left-hand corner of the web page.

28. Using the search box located under AWS services, enter VPC.

29. Click the VPC search result.The VPC dashboard is opened.

Proprietary Information of Altair Engineering

Page 149: Control Administrator's Guide - Altair PBS Works · Altair Control 2019.1 Administrator's Guide iv • Altair Embed SE™ ©1989-2019 (formerly solidThinking Embed® SE) • Altair

Control Administrator's Guide9 Cloud Bursting p.144

30. In the menu located on the left-hand side of the web page, click Subnets.

31. Select the subnet for the bursting VPC by enabling the check box next to its name.

32. In the Description tab located at the bottom of the web page, hover over the Subnet ID.

A copy to clipboard icon is displayed.

33. Click .

34. Navigate to the Control browser window.

35. For the Subnet ID, paste the Subnet ID copied from AWS.

36. Enable the Add public IP to VMs if you need to connect to the cloud bursting nodes from theinternet.

It is recommended to leave this disabled.

37. For Cloud-init script, click Browse, browse for the script file and select it.

This script will run after a cloud node is burst.

38. Add public SSH keys for any users that you want to have access to the burst nodes:

a) Login to the PBS Professional Server.

b) Copy the public SSH key for the user.

Public key files are usually stored in /.ssh in the user's home directory.

c) Navigate to the Control browser window.

d) Click Add to add the public key.

e) For Public SSH keys, paste the public SSH key.

f) Repeat steps a through e to add additional public SSH keys.

The public keys are copied to each burst node for secure connectivity.

39. Click Instantiate scenario.

Proprietary Information of Altair Engineering

Page 150: Control Administrator's Guide - Altair PBS Works · Altair Control 2019.1 Administrator's Guide iv • Altair Embed SE™ ©1989-2019 (formerly solidThinking Embed® SE) • Altair

Control Administrator's Guide9 Cloud Bursting p.145

9.6 Configuring Microsoft Azure Cloud BurstingCreate the necessary components in Microsoft Azure that are required for cloud bursting and thencreate a Control cloud bursting scenario.

Before you can begin creating the Azure components necessary for cloud bursting, you must purchasean Azure subscription and have created a tenant. For more information about subscriptions see What isan Azure subscription. For more information about tenants How to get an Azure Active Directory tenant.As a final step, add users to the Active Directory.

9.6.1 Registering the Control Application with AzureRegister Control with Azure so that you can later create a cloud account in Control.

You must have already purchased an Azure subscription, obtained a tenant and have an Azure useraccount.

Azure documentation can be found at Integrating applications with Azure Active Directory

Register the Control web application with the Azure Active Directory and create an API access key.Manage access to the newly create application by assigning a role. Also, obtain your Azure tenant IDand subscription ID. Once this task is complete, you should have the following information stored in afile:

• Application ID generated when registering Control with the Azure Active Directory.

• API access key generated for the newly registered Control application.

• Subscription ID

• Tenant ID

This information is necessary to create a cloud account in Control.

1. Login in to your Microsoft Azure account.

Register Control to the Azure Active Directory.

2. In the menu located on the left-hand side of the web page, click Azure Active Directory.

3. Under MANAGE, click App registrations.

Proprietary Information of Altair Engineering

Page 151: Control Administrator's Guide - Altair PBS Works · Altair Control 2019.1 Administrator's Guide iv • Altair Embed SE™ ©1989-2019 (formerly solidThinking Embed® SE) • Altair

Control Administrator's Guide9 Cloud Bursting p.146

Figure 28: Application Registration

4. Click New application registration.

5. Enter the following to register Control to the Azure Active Directory:

a) For Name, enter the name of the application.

The name can be anything meaningful to your organization, e.g., pc_cloudapp.

a) For Application type, select Web app / API.

b) For Sign-on URL, enter the URL https://hostname:9500/pc.

where hostname is the hostname of the machine where the Control Web Interface isinstalled. This is the URL that is used to login to Control.

6. Click Create.Once the application registration is complete, its details are displayed, including an Application ID.

Obtain the application id of the newly registered application.

7. Hover over the Application ID and click when it appears.

8. Store the Application ID to a file.

The Application ID is used later to create a cloud account in Control.

Create an api-key for your newly registered application.

9. Click .

Figure 29: Registered Application Settings

10. Under API Access, click Keys.

11. Enter the following to create an api-key:

Proprietary Information of Altair Engineering

Page 152: Control Administrator's Guide - Altair PBS Works · Altair Control 2019.1 Administrator's Guide iv • Altair Embed SE™ ©1989-2019 (formerly solidThinking Embed® SE) • Altair

Control Administrator's Guide9 Cloud Bursting p.147

a) For DESCRIPTION, enter pc_access_key.

b) For EXPIRES, select Never Expires.

12. Click Save.A key value is generated and is displayed under the heading VALUE.

13. Copy the key value.

14. Store the key value to a file.

The key value is used later to create a cloud account in Control.

Obtain your Azure subscription ID.

15. In the menu located on the left-hand side of the web page, click Cost Management + Billing.

16. Under BILLING, click Subscriptions.

Figure 30: Subscriptions

17. Locate and click your subscription.Details about the subscription are displayed, including a Subscription ID.

18. Hover over the Subscription ID and click when it appears.

19. Store the Subscription ID value to a file.

The Subscription ID is used later to create a cloud account in Control.

Assign an access control role to the newly registered application.

20. Click Access control (IAM).

Figure 31: Add Access Controls

21. Click Add.

22. Click Add role assignment.23. In the Add permissions panel, enter the following to assign a role to the newly registered

application:

Proprietary Information of Altair Engineering

Page 153: Control Administrator's Guide - Altair PBS Works · Altair Control 2019.1 Administrator's Guide iv • Altair Embed SE™ ©1989-2019 (formerly solidThinking Embed® SE) • Altair

Control Administrator's Guide9 Cloud Bursting p.148

a) For Role, select Contributor.b) For Assign access to, select Azure AD user, group, or service principal.c) For Select, search for the newly registered application by entering its name, e.g.,

pc_cloudapp.

d) Select the application by clicking on it.

24. Click Save.

Obtain your Azure tenant ID:

25. At the top of the web page, click ?.

26. Click Show diagnostics.

A dialog box is displayed allowing a file called PortalDiagnostics.json to be saved.

27. Open the file using any text editor.

28. Search for tenantId.

29. Store the value of tenantId to a file.

The Tenant ID is used later to create a cloud account in Control

9.6.2 Add a Microsoft Azure Cloud AccountAdd an Azure cloud account to Control so that you can create a cloud bursting scenario or an appliance.

You should have the following information stored in a file:

• Application ID generated when registering Control with the Azure Active Directory.

• API access key generated for the newly registered Control application.

• Subscription ID

• Tenant ID

This information is necessary to create a cloud account in Control. If you do not have this informationsee Registering the Control Application with Azure.

1. Login to Control.

2. Click the Cloud tab.

3. Under Infrastructure, click Cloud.

4. Click .

5. Enter the following to add a cloud account:

a) For Account name, enter any name for the cloud account.

The name can be anything meaningful to your organization, e.g., azure_cloudaccount.

b) For Client ID, enter the Application ID generated when Control was registered with the AzureActive Directory.

c) For Secret Key, enter the API access key generated for the newly registered Controlapplication.

d) For AD tenant ID, enter your Azure tenant ID.

e) For Subscription ID, enter your Azure subscription ID.

Proprietary Information of Altair Engineering

Page 154: Control Administrator's Guide - Altair PBS Works · Altair Control 2019.1 Administrator's Guide iv • Altair Embed SE™ ©1989-2019 (formerly solidThinking Embed® SE) • Altair

Control Administrator's Guide9 Cloud Bursting p.149

6. Click Create account.

9.6.3 Creating a Resource GroupCreate a resource group that will contain a virtual network and a virtual machine for cloud bursting.

You must have already purchased an Azure subscription and have an Azure user account.

Azure documentation can be found at What is an Azure resource group and Manage Azure resourcesthrough portal.

A resource group is container that holds related resources for an Azure solution. The resource group caninclude all the resources for the solution, or only those resources that you want to manage as a group.You decide how you want to allocate resources to resource groups based on what makes the most sensefor your organization. Once the resource group is created, resources that are placed into the resourcegroup are a virtual network, a virtual machine, and an image of the virtual machine.

1. Login to your Microsoft Azure account.

2. In the menu located on the left-hand side of the web page, click Resource groups.

3. Click + Add.

Enter the following to configure the basic settings for the resource group:

4. For PROJECT DETAILS enter the following:

a) For Subscription, choose the subscription to be billed for the use of the VM.

b) For Resource group, enter a name for the resource group.

The name can be anything meaningful to your organization, e.g., bursting_resource_group.

5. For RESOURCE DETAILS enter the following:

a) For Region, select a location based on the geographical location of users.

6. Click Review + create.

7. Click Create.

It may take a moment to create the resource group. All resources (networks, virtual machines,etc.) that are created are placed within this resource group. The name of the resource group isrequired for creating a bursting scenario in Control.

9.6.4 Creating a Virtual NetworkCreate a virtual network so that you can later create a virtual machine for cloud bursting.

You must have already created a resource group.

Azure documentation can be found at Virtual Network Documentation.

1. Login to your Microsoft Azure account.

2. In the menu located on the left-hand side of the web page, click Virtual networks.

3. Click +Add.

4. For Name, enter a name for the virtual network.

The name can be anything meaningful to your organization, e.g., bursting_virtual_network

Proprietary Information of Altair Engineering

Page 155: Control Administrator's Guide - Altair PBS Works · Altair Control 2019.1 Administrator's Guide iv • Altair Embed SE™ ©1989-2019 (formerly solidThinking Embed® SE) • Altair

Control Administrator's Guide9 Cloud Bursting p.150

5. For Address space, enter an address range for the network using CIDR notation.

6. For Subscription, select the same subscription as was selected for the previously createdresource group.

7. For Resource group, select the previously created resource group.

8. For Location, select the same geographical location as was selected for the previously createdresource group.

9. For Subnet, enter the following:

a) For Name, enter a name for the virtual machine's subnet.

The name can be anything meaningful to your organization, e.g., bursting_subnet

b) For Address range, enter an address range for the subnet in CIDR notation.

10. Click Create.

It may take a moment to create the virtual network. The name of the virtual network is requiredfor creating a bursting scenario in Control.

9.6.5 Creating a Virtual MachineCreate a CentOS 7.3 virtual machine in Azure.

You must have already created a virtual network.

You may want to view the following web page to learn about Azure Managed Disks before creating aVM. Additionally, a video is available from Microsoft that shows how to create a virtual machine: Createa Linux Virtual Machine.

1. Login to your Microsoft Azure account.

2. In the menu located on the left-hand side of the web page, click Virtual Machines.

3. Click Add.

Enter the following to configure the basic settings for the virtual machine:

4. For INSTANCE DETAILS enter the following:

a) For Subscription, choose the subscription to be billed for the use of the VM.

b) For Resource group, choose the previously created resource group.

c) For Virtual machine name, enter a name for the virtual machine.

The name can be anything meaningful to your organization, e.g., bursting-vm.

d) For Region, select the same geographical location as was selected for the previously createdresource group.

e) For Availability options, choose No infrastructure redundancy required.

f) For Image, click the Browse all images link.

g) In the search bar, enter CentOS-based 7.3.

h) Select the CentOS-based 7.3 HPC option from the search results list.

i) For Size, click the Change size link and select a machine size appropriate for your site'sworkload based on:

• the number of cores

Proprietary Information of Altair Engineering

Page 156: Control Administrator's Guide - Altair PBS Works · Altair Control 2019.1 Administrator's Guide iv • Altair Embed SE™ ©1989-2019 (formerly solidThinking Embed® SE) • Altair

Control Administrator's Guide9 Cloud Bursting p.151

• the amount of memory

• storage

• network performance

Consider the nature of the applications that you plan to deploy on the instance, the numberof users that you expect to use the applications, and also how you expect the load to scale inthe future. Remember to also factor in the CPU and memory resources that are necessary forthe operating system.

j) Click Select.5. For ADMINISTATOR ACCOUNT, enter the following:

a) For Authentication type, enable SSH public key.

b) For Username, enter a username of a user account that exists on your site's PBS Server.

c) For SSH public key, copy the SSH public key (i.e., id_rsa.pub) of the user account and pasteit.

This user will have sudo rights and will be able to SSH into the VM to install the PBS MoM.

6. For INBOUND PORT RULES, enter the following:

a) For Public inbound ports, enable Allow selected ports.

b) For Select inbound ports, enable SSH (22).

Enter the following to configure the storage settings for the virtual machine:

7. Scroll back up to the top of the webpage and click the Disks tab.

8. For DISK OPTIONS, enter the following:

a) For OS disk size, choose an appropriate disk size based on your site's needs.

b) For OS disk type, choose one of the following options:

• Premium SSD

• Standard SSD

• Standard HDD

Choose SSD for I/O-intensive applications, where low latency and high throughput arecritical. For testing, consider HDD to keep costs down, as you scale up and down quickly.

9. For ADVANCED, enter the following:

a) Click ADVANCED.

b) For Use managed disks, choose one of the following options:

• Yes to use managed disks.

• No to not use managed disks.

Enable this feature to have Azure automatically manage the availability of disks to providedata redundancy and fault tolerance, without creating and managing storage accounts onyour own. This option is recommended by Azure as it is a lot more scalable.

Enter the following to configure the networking settings for the virtual machine:

10. Click the Networking tab.

11. For CONFIGURE VIRTUAL NETWORKS, enter the following:

a) For Virtual network, choose the virtual network previously created.

Proprietary Information of Altair Engineering

Page 157: Control Administrator's Guide - Altair PBS Works · Altair Control 2019.1 Administrator's Guide iv • Altair Embed SE™ ©1989-2019 (formerly solidThinking Embed® SE) • Altair

Control Administrator's Guide9 Cloud Bursting p.152

12. Click Review + create.

13. Click Create.

It may take a few minutes for the VM to be deployed. You will use this virtual machine to createan OS image.

Once the virtual machine is deployed a message is displayed indicating success, click on Go toresource.

9.6.6 Installing and Configuring the PBS MoM on the VMInstall the PBS MoM on the virtual machine and configure the VM per your site's environment.

A cloud-init script is provided during the creation of the bursting scenario. This script is ran after acloud node is burst and is used to set up the system environment so that nodes burst in the cloud cancommunicate with the PBS Professional Server, therefore installation of cloud-init on the VM is required.For more information about installing cloud-init, refer to the Cloudinit Documentation.

1. Login to your site's PBS Server as the user account (username and the public SSH key) providedduring the creation of the VM.

2. SSH into the virtual machine using the public IP address of the VM:ssh IPV4PublicIP

where IPV4PublicIP is the public IP address of the virtual machine.

Figure 32: Bursting Virtual Machine

3. Enter the command:sudo -i

4. Using the PBS Professional Installation and Upgrade Guide, install and configure the PBS MOM.

5. Once the PBS MOM is installed, edit the /etc/pbs.conf file.

6. Change the value of PBS_SERVER to the hostname of the PBS Server.

7. Save the file.

8. Edit the file PBS_HOME/mom_priv/config where PBS_HOME is the home directory of PBS.

9. Change the value of $clienthost to the hostname of the PBS Server.

Proprietary Information of Altair Engineering

Page 158: Control Administrator's Guide - Altair PBS Works · Altair Control 2019.1 Administrator's Guide iv • Altair Embed SE™ ©1989-2019 (formerly solidThinking Embed® SE) • Altair

Control Administrator's Guide9 Cloud Bursting p.153

10. Save the file.

11. Configure the VM for your site's environment such as pointing it the DNS server, mounting filesystems, connecting it to the authentication service, installing any applications, etc.

12. Install cloud-init.

13. Connect the VM to your PBS server and submit a job to ensure it is working as expected.

9.6.7 Creating an OS ImageCreate an image of the virtual machine configured per your site's requirements.

You must have already create a virtual machine, installed the PBS MoM on the VM and configured theVM per your site's environment.

To create an OS image requires the Azure CLI. Refer to these instructions for installing the CLI How toinstall the Azure CLI. It is recommended to install the CLI on a Windows or Mac machine and then usethe command prompt to execute the CLI commands.

Before you can create an OS image of the previously created VM, you must first SSH into the VM anddeprovision it. Next you will use the Azure CLI to deallocate and generalize the VM and then create theimage. Generalizing the virtual machine removes any SSH keys and DNS settings from the VM.

Follow Step 1 and Step 2 as documented at the URL How to create an image of a virtual machine orVHD to create an image of the VM.

Note: Before you can deallocate the virtual machine you may have to execute the followingcommands to set your subscription to be the active subscription.az account listaz account set --subscription yourSubscriptionID

An OS image is created of the virtual machine.

You can now delete the virtual machine so that you are no longer charged for it.

9.6.8 Create an Azure Cloud Bursting ScenarioCreate an Azure bursting scenario so that PBS execution nodes can be burst in the cloud.

Open a browser window and login to your Microsoft Azure account. Open a second browser window andlogin to Control.

Before creating a cloud bursting scenario, read the following Azure article about scale sets What arevirtual machine scale sets.

A cloud-init script is provided during the creation of the bursting scenario. This script is ran after acloud node is burst and is used to set up the system environment so that nodes burst in the cloud cancommunicate with the PBS Professional Server. For more information about writing cloud-init scripts,refer to the Cloudinit Documentation and the Cloud Bursting chapter of the ControlAdministrator'sGuide.

Proprietary Information of Altair Engineering

Page 159: Control Administrator's Guide - Altair PBS Works · Altair Control 2019.1 Administrator's Guide iv • Altair Embed SE™ ©1989-2019 (formerly solidThinking Embed® SE) • Altair

Control Administrator's Guide9 Cloud Bursting p.154

Note: To create a bursting scenario you must enter a resource group, however theresources necessary for creating the bursting scenario (network, subnet, network securitygroup and the OS image) can all reside in a different resource group. However, the resourcegroups must be in the same geographic location for this to work.

1. Navigate to the Control browser window.

2. Click Cloud.

3. Under Infrastructure, click Bursting.

4. Click Add Bursting Scenario.

5. Enter information about the bursting scenario such as name, cloud provider, cloud account, andgeographic location.

a) For Name, enter a name for the cloud bursting scenario.

b) For Description, enter a description.

c) For Cloud Account, choose a previously created Azure cloud account from the list.

d) For Select a region, select the same geographic location as was selected for the previouslycreated bursting resource group.

6. Click Next.Enter information about the Azure resources necessary to burst in the cloud.

7. For Domain name, provide the domain name of your organization for the cloud bursting network.altair.com, burst.altair.com

8. For Hostname prefix, enter a prefix for the names of the burst nodes.If "node" is entered , the cloud bursting nodes will be available to PBS Professional as node1,node2, node3, etc.

9. Navigate to the Azure browser window.

10. In the menu located on the left-hand side of the web page, click Resource Groups.

11. Copy the name of the resource group created for cloud bursting.

12. Navigate to the Control browser window.

13. For Resource group name, paste the resource group copied from Azure.

14. Navigate to the Azure browser window.

15. In the menu located on the left-hand side of the web page, click Virtual Networks.

16. Copy the name of the virtual network created for cloud bursting.

17. Navigate to the Control browser window.

18. For Network name, paste the name of the virtual network copied from Azure.

If the network is contained in a resource group that is different from the one entered for thebursting scenario, it must be entered as Resource Group Name/Virtual Network Name.

19. Navigate to the Azure browser window.

20. Select the virtual network created for cloud bursting.

21. Click Subnets.

Proprietary Information of Altair Engineering

Page 160: Control Administrator's Guide - Altair PBS Works · Altair Control 2019.1 Administrator's Guide iv • Altair Embed SE™ ©1989-2019 (formerly solidThinking Embed® SE) • Altair

Control Administrator's Guide9 Cloud Bursting p.155

Figure 33: Subnet

22. For Subnet name, copy the name of the subnet created for the cloud bursting virtual network.

23. Navigate to the Control browser window.

24. For Subnet name, paste the Subnet copied from Azure.

If the subnet is contained in a resource group that is different from the one entered for thebursting scenario, it must be entered as Resource Group Name/Subnet Name.

25. Navigate to the Azure browser window.

26. In the menu located on the left-hand side of the web page, click Resource Groups.

27. Select the Resource Group created for cloud bursting.

28. In the list, locate the Network Security group created for cloud bursting.

The Type is Network Security group.

29. Copy the name of the network security group.

30. Navigate to the Control browser window.

31. For Network security group name, paste the network security group copied from Azure.

If the network security group is contained in a resource group that is different from the oneentered for the bursting scenario, it must be entered as Resource Group Name/NetworkSecurity Group Name.

32. Choose one of the following options based on whether you chose to use the Azure managed diskfeature when creating the virtual machine:

• Enable Managed Storage if you enabled the managed disk feature.

• Disable Managed Storage if you disabled the managed disk feature.

33. Navigate to the Azure browser window.

34. In the menu located on the left-hand side of the web page, click Resource Groups.

35. Select the Resource Group created for cloud bursting.

36. In the list, locate the image that was created for cloud bursting.

The Type is Image.

37. Select the image.

Proprietary Information of Altair Engineering

Page 161: Control Administrator's Guide - Altair PBS Works · Altair Control 2019.1 Administrator's Guide iv • Altair Embed SE™ ©1989-2019 (formerly solidThinking Embed® SE) • Altair

Control Administrator's Guide9 Cloud Bursting p.156

38. Choose one of the following options:

• If you chose to use managed disks when you created the VM, copy the name of the image.

• If you did not choose to use managed disks when you created the VM, copy the Linux SourceBLOB URI.

39. Navigate to the Control browser window.

40. For OS image, paste the information copied from Azure.

If the OS image is contained in a Resource group that is different from the one entered for thebursting scenario, it must be entered as Resource Group Name/OS Image Name or ResourceGroup Name/URI.

41. Enable the Add public IP to VMs if you need to connect to the cloud bursting nodes from theinternet.

It is recommended to leave this disabled.

42. For Maximum number of VMs inside a ScaleSet, specify the limit for your cloud bursting scenario.

If you use InfiniBand, Azure limits the number of VMs on a scaleset to 100. Without InfiniBand, fora scaleset with managed disk and custom image, you can specify a higher limit.

43. For Cloud-init script, click Browse, browse for the script file and add it.

This script will run after a cloud node is burst.

44. Add public SSH keys for any users that you want to have access to the burst nodes:

a) Login to the PBS Server.

b) Copy the public SSH key for the user.

Public key files are usually stored in /.ssh in the user's home directory.

c) Navigate to the Control browser window.

d) Click Add to add the public key.

e) For Public SSH keys, paste the public SSH key.

f) Repeat steps a through e to add additional public SSH keys.

The public keys are copied to each burst node for secure connectivity.

45. Click Instantiate scenario.

Proprietary Information of Altair Engineering

Page 162: Control Administrator's Guide - Altair PBS Works · Altair Control 2019.1 Administrator's Guide iv • Altair Embed SE™ ©1989-2019 (formerly solidThinking Embed® SE) • Altair

Control Administrator's Guide9 Cloud Bursting p.157

9.7 Configure Google Cloud Platform Cloud BurstingCreate the necessary components in GCP that are required for cloud bursting and then create a Controlcloud bursting scenario.

9.7.1 Create a ProjectCreate a GCP Project so that you can create the necessary cloud components for cloud bursting.

GCP documentation can be found at Creating and Managing Projects.

Google Cloud Platform projects form the basis for creating, enabling, and using all Cloud Platformservices including managing APIs, enabling billing, adding and removing collaborators, and managingpermissions for Cloud Platform resources.

1. Login to the GCP console.

2. Click located in the upper left-hand corner of the web page.

3. Click Home.

4. Click Create.

5. For Project Name, enter a name for the project.

The name can be anything meaningful to your organization, e.g., pc_cloudproject.

6. Click Create.

It may take a few moments to create the project.

7. Using a browser, navigate to the following URL: https://console.developers.google.com/apis/library/compute.googleapis.com?project=PROJECTNAME

where PROJECTNAME is the name of the project.

8. Click Enable.

9.7.2 Create a Service AccountCreate a Google Cloud Platform service account so that you can later create a cloud account in Control.

You must already have signed up for an GCP account.

GCP documentation can be found at Understanding Service Accounts and Compute Engine IAM Roles .

A service account is a special Google account that belongs to an application or a virtual machine,instead of to an individual end user. Create a service account and give this account administrativepermissions for all Compute Engine resources. Once this task is complete, you should have downloadeda JSON file containing the following:

• Project ID

• Client ID

• Client email

• Private key ID

Proprietary Information of Altair Engineering

Page 163: Control Administrator's Guide - Altair PBS Works · Altair Control 2019.1 Administrator's Guide iv • Altair Embed SE™ ©1989-2019 (formerly solidThinking Embed® SE) • Altair

Control Administrator's Guide9 Cloud Bursting p.158

• Private key

This information is necessary to create a cloud account in Control.

1. Login to the GCP console.

2. Click located in the upper left-hand corner of the web page.

3. Click IAM & admin > Service accounts.

4. Click CREATE SERVICE ACCOUNT.

5. Enter the following to create a service account:

a) For the Service account name, enter a name for the service account.

The name can be anything meaningful to your organization, e.g., pc-service-account.

b) Click CREATE.

c) For the Project role, click Select a role > Compute Engine > Compute Admin.

This role gives full control of all Compute Engine resources.

d) Click CONTINUE.

e) Under Create key (optional), click CREATE KEY.

f) For Key type, enable JSON.

g) Click CREATE.

h) Save the JSON file in a secure location.

The information in this file is used later to add a cloud account to Control.

i) Click CLOSE.

j) Click DONE.The service account is created.

9.7.3 Add a GCP Cloud AccountAdd an existing GCP service account to Control so that you can create a cloud bursting scenario or anappliance.

You must have created a GCP service account, given it Compute Admin permissions and downloadeda JSON file containing information about the service account. The information in the JSON file isnecessary to create a cloud account in Control. If you do not have this information see Create a ServiceAccount.

1. Login to Control.

2. Click the Cloud tab.

3. Under Infrastructure, click Cloud.

4. Click .

5. Enter the following to add a cloud account:

a) For Account name, enter any name for the cloud account.

The name can be anything meaningful to your organization, e.g., gcp_cloudaccount.

b) For Project ID, copy and paste the project_id value from the JSON file.

Proprietary Information of Altair Engineering

Page 164: Control Administrator's Guide - Altair PBS Works · Altair Control 2019.1 Administrator's Guide iv • Altair Embed SE™ ©1989-2019 (formerly solidThinking Embed® SE) • Altair

Control Administrator's Guide9 Cloud Bursting p.159

c) For Client ID, copy and paste the client_id value from the JSON file.

d) For Client Mail, copy and paste the client_email value from the JSON file.

e) For Private Key ID, copy and paste the private_key_id value from the JSON file.

f) For Private Key, copy and paste the private_key value from the JSON file.

Important: Do not copy the \n characters at the end of the private_key value.

6. Click Create account.

9.7.4 Create a Virtual Private Cloud NetworkCreate a Virtual Private Network (VPC) in GCP so that you can later create a virtual machine for cloudbursting.

GCP documentation can be found at Virtual Private Cloud Documentation and Using VPC Networks .

1. Login to the GCP console.

2. Click located in the upper left-hand corner of the web page.

3. Click VPC network > VPC networks.

4. Click CREATE VPC NETWORK.

5. Enter the following to create a VPC:

a) For the Name, enter a name for the VPC.

The name can be anything meaningful to your organization, e.g., bursting-vpc.

b) In the Subnets section, click the Custom tab under Subnet creation mode.

c) For Name, enter a name for the subnet.

The name can be anything meaningful to your organization, e.g., bursting-subnet.

d) For Region, select a Region based on the geographical location of users.

e) For IP address range, enter an IP address range using CIDR notation

f) For Private Google access, enable Off.g) Click Done.

h) For Dynamic routing mode, enable Regional.6. Click Create.

Creating the VPC network may take some time. Do not proceed until the VPC is created.

7. Select the VPC by clicking on its name.

8. Click the Firewall rules tab.

9. Click CREATE FIREWALL RULE.

10. Enter the following to create a firewall rule:

a) For Name, enter a name for the firewall rule.

The name can be anything meaningful to your organization, e.g., ssh-all.

b) For Direction of Traffic, enable Ingress.

c) For Action on match, enable Allow.

Proprietary Information of Altair Engineering

Page 165: Control Administrator's Guide - Altair PBS Works · Altair Control 2019.1 Administrator's Guide iv • Altair Embed SE™ ©1989-2019 (formerly solidThinking Embed® SE) • Altair

Control Administrator's Guide9 Cloud Bursting p.160

d) For Targets, select All instances in the network.

e) For Source filter, select IP ranges.

f) For Source IP ranges, enter 0.0.0.0/0.

g) For Protocols and ports, enable Specified protocols and ports.

h) Enable tcp.

i) Enter 22.

11. Click Create.

9.7.5 Creating a Virtual MachineCreate a CentOS 7.3 virtual Machine in GCP.

You must have already created a virtual network with a subnet and added a firewall rule.

GCP documentation can be found at Virtual Machine Instances and Creating and Starting a VM Instance.

1. Login to the GCP console.

2. Click located in the upper left-hand corner of the web page.

3. Click Compute Engine > VM instances.

4. Click CREATE INSTANCE.

5. Enter the following to create a virtual machine:

a) For the Name, enter a name for the virtual machine.

The name can be anything meaningful to your organization, e.g., bursting-vm.

b) For Zone, select a zone that is in the same Region as the subnet of the previously createdVPC.

c) In the Machine type, click the Customize link.

Figure 34: Customize the Machine Type

d) Specify the CPUs, GPUs and RAM.

Consider the nature of the applications that you plan to deploy on the instance, the numberof users that you expect to use the applications, and also how you expect the load to scale inthe future. Remember to also factor in the CPU and memory resources that are necessary forthe operating system.

e) For Boot disk, click Change.

f) Choose CentOS 7.

g) For Boot disk type, choose one of the following options:

• Standard persistent disk

Proprietary Information of Altair Engineering

Page 166: Control Administrator's Guide - Altair PBS Works · Altair Control 2019.1 Administrator's Guide iv • Altair Embed SE™ ©1989-2019 (formerly solidThinking Embed® SE) • Altair

Control Administrator's Guide9 Cloud Bursting p.161

• SSD persistent disk

Choose SSD for I/O-intensive applications, where low latency and high throughput arecritical. For testing, consider Standard persistent disk to keep costs down.

h) For Size, specify the size of the boot disk.

i) Click Select.j) Under Identity and API access, for Service Account, select No service account.k) For Firewall, choose Allow HTTP traffic.

l) Click Management, disks, networking, SSH Keys.

m) Click the Networking tab.

n) Click Add network interface.

o) For Network, choose the VPC you previously created for bursting.

p) For Network Service Tier, click Standard.

q) Click Done.

r) Delete any default network interfaces that might have been automatically generated.

s) Click the Security tab.

t) For SSK Keys, copy the SSH public key (i.e., id_rsa.pub) of a user account that exists on yoursite's PBS Server and paste it.

This user will have sudo rights and will be able to SSH into the VM to install the PBS MoM.

6. Click Create.Creating the virtual machine may take some time.

9.7.6 Installing and Configuring the PBS MoM on the VMInstall the PBS MoM on the virtual machine and configure the VM per your site's environment.

The user name for logging into the virtual machine is determined by the SSH key that was providedduring the creation of the virtual machine.

A cloud-init script is provided during the creation of the bursting scenario. This script is ran after acloud node is burst and is used to set up the system environment so that nodes burst in the cloud cancommunicate with the PBS Professional Server, therefore installation of cloud-init on the VM is required.For more information about installing cloud-init, refer to the Cloudinit Documentation.

1. Login to your site's PBS Server as the user account (public SSH key) provided during the creationof the VM.

2. SSH into the virtual machine using the public IP address of the VM:ssh IPV4PublicIP

where IPV4PublicIP is the public IP address of the virtual machine.

Proprietary Information of Altair Engineering

Page 167: Control Administrator's Guide - Altair PBS Works · Altair Control 2019.1 Administrator's Guide iv • Altair Embed SE™ ©1989-2019 (formerly solidThinking Embed® SE) • Altair

Control Administrator's Guide9 Cloud Bursting p.162

Figure 35: Bursting Virtual Machine

3. Enter the command:sudo -i

4. Using the PBS Professional Installation and Upgrade Guide, install and configure the PBS MOM.

5. Once the PBS MOM is installed, edit the /etc/pbs.conf file.

6. Change the value of PBS_SERVER to the hostname of the PBS Server.

7. Save the file.

8. Edit the file PBS_HOME/mom_priv/config.

where PBS_HOME is the home directory of PBS.

9. Change the value of $clienthost to the hostname of the PBS Server.

10. Save the file.

11. Configure the VM for your site's environment such as pointing it the DNS server, mounting filesystems, connecting it to the authentication service, installing any applications, etc.

12. Install cloud-init.

13. Connect the VM to your PBS server and submit a job to ensure it is working as expected.

9.7.7 Create an OS ImageCreate an image of the virtual machine configured per your site's requirements.

You must have already create a virtual machine, installed the PBS MoM on the VM and configured theVM per your site's environment.

GCP documentation can be found at Creating, Deleting, and Deprecating Custom Images.

1. Login to the GCP console.

2. Click located in the upper left-hand corner of the web page.

3. Click Compute Engine > VM instances.A list of virtual machines is displayed.

4. Click next to the virtual machine that was created for cloud bursting.

Figure 36: Bursting Virtual Machine

5. Click Stop.

It may take some time for the VM to be stopped. Do not proceed until the VM is stopped.

Proprietary Information of Altair Engineering

Page 168: Control Administrator's Guide - Altair PBS Works · Altair Control 2019.1 Administrator's Guide iv • Altair Embed SE™ ©1989-2019 (formerly solidThinking Embed® SE) • Altair

Control Administrator's Guide9 Cloud Bursting p.163

6. In the menu located on the left-hand side of the web page, click Images .

7. Click CREATE IMAGE.

8. Enter the following to create an image:

a) For Name, enter a name for the image.

The name can be anything meaningful to your organization, e.g., bursting-image.

b) For Source select Disk.

c) For Source disk, select the previously created virtual machine.

9. Click Create.

It may take some time to create the image. Do not proceed until the image is created.

You can now delete the virtual machine so that you are no longer charged for it.

9.7.8 Create a GCP Cloud Bursting ScenarioCreate an GCP bursting scenario so that PBS execution nodes can be burst in the cloud.

Open a browser window and login to your GCP console. Open a second browser window and login toControl.

A cloud-init script is provided during the creation of the bursting scenario. This script is ran after acloud node is burst and is used to set up the system environment so that nodes burst in the cloud cancommunicate with the PBS Professional Server. For more information about writing cloud-init scripts,refer to the Cloudinit Documentation and the Cloud Bursting chapter of the ControlAdministrator'sGuide.

1. Navigate to the Control browser window.

2. Click Cloud.

3. Under Infrastructure, click Bursting.

4. Click Add Bursting Scenario.

5. Enter information about the bursting scenario such as name, cloud provider, cloud account, andgeographic location.

a) For Name, enter a name for the cloud bursting scenario.

b) For Description, enter a description.

c) For Cloud Account, choose a previously created GCP cloud account from the list.

d) For Select a region, select the same geographic location as was selected for the previouslycreated bursting VPC.

6. Click Next.Enter information about the GCP resources necessary to burst in the cloud.

7. For Domain name, provide the domain name of your organization for the cloud bursting network.altair.com, burst.altair.com

8. For Hostname prefix, enter a prefix for the names of the burst nodes.If "node" is entered , the cloud bursting nodes will be available to PBS Professional as node1,node2, node3, etc.

Proprietary Information of Altair Engineering

Page 169: Control Administrator's Guide - Altair PBS Works · Altair Control 2019.1 Administrator's Guide iv • Altair Embed SE™ ©1989-2019 (formerly solidThinking Embed® SE) • Altair

Control Administrator's Guide9 Cloud Bursting p.164

9. Navigate to the GCP browser window.

10. Click located in the upper left-hand corner of the web page.

11. Click VPC network > VPC networks.

12. Click on the name of the VPC that was created for cloud bursting.VPC network details are displayed.

13. Copy the name of the VPC network.

14. Navigate to the Control browser window.

15. For Network name, paste the name of the VPC network copied from GCP.

16. Navigate to the GCP browser window.

17. Copy the name of the VPC network subnet.

18. Navigate to the Control browser window.

19. For Subnetwork name, paste the name of the VPC network subnet copied from GCP.

20. Enable the Add public IP to VMs if you need to connect to the cloud bursting nodes from theinternet.

It is recommended to leave this disabled.

21. Navigate to the GCP browser window.

22. Click located in the upper left-hand corner of the web page.

23. Click Compute Engine > Images.

24. Select the image created for cloud bursting.

25. Click Equivalent REST26. Copy the value for the entry called selfLink.

27. Navigate to the Control browser window.

28. For OS Image URI, paste the link copied from GCP.

29. For Cloud-init script, click Browse, browse for the script file and add it.

This script will run after a cloud node is burst.

30. Add public SSH keys for any users that you want to have access to the burst nodes:

a) Login to the PBS Server.

b) Copy the public SSH key for the user.

Public key files are usually stored in /.ssh in the user's home directory.

c) Navigate to the Control browser window.

d) Click Add to add the public key.

e) For Public SSH keys, paste the public SSH key.

f) Repeat steps a through e to add additional public SSH keys.

The public keys are copied to each burst node for secure connectivity.

31. Click Instantiate scenario.

Proprietary Information of Altair Engineering

Page 170: Control Administrator's Guide - Altair PBS Works · Altair Control 2019.1 Administrator's Guide iv • Altair Embed SE™ ©1989-2019 (formerly solidThinking Embed® SE) • Altair

Control Administrator's Guide9 Cloud Bursting p.165

9.8 Configure Oracle Cloud Platform Cloud BurstingCreate the necessary components in Oracle Cloud that are required for cloud bursting and then create aControl cloud bursting scenario.

9.8.1 Create an Oracle Cloud User AccountCreate an Oracle Cloud user account so that you can later create a cloud account in Control.

You must already have signed up for an Oracle Cloud account and have an associated tenancy.

Oracle documentation can be found at Adding Users and Resource Identifiers.

An Oracle user account must be created to add an Oracle cloud account to Control. Once this task iscomplete, you should have the following information stored in a file:

• The user account's Oracle Cloud ID (OCID).

This information is necessary to create a cloud account in Control.

1. Login to the Oracle Cloud Infrastructure console.

2. Click located in the upper left-hand corner of the web page.

3. Click Identity > Users.

4. Click Create User.5. Enter the following to create the user:

a) For NAME, enter a name for the user.

The name can be anything meaningful to your organization, e.g., pc_clouduser.

b) For DESCRIPTION, enter a description of the user.

6. Click Create.The user account is created and displayed in the users list.

7. Click Show located under the name of the user.The user account's OCID is displayed.

8. Click Copy to copy the OCID.

9. Store the OCID to a file.

The OCID is used later to create a cloud account in Control.

10. Click Group from the menu located on the left-hand side of the web page.

11. Click Create Group.

12. Enter the following:

a) For Name, enter Administrators.

b) For Description, enter a description for the group.

c) Click Submit.The group is created and is displayed in the Groups list.

13. Click on the name of the group.

14. Click Add User to Group.

Proprietary Information of Altair Engineering

Page 171: Control Administrator's Guide - Altair PBS Works · Altair Control 2019.1 Administrator's Guide iv • Altair Embed SE™ ©1989-2019 (formerly solidThinking Embed® SE) • Altair

Control Administrator's Guide9 Cloud Bursting p.166

a) For User, select the user that was previously created (e.g., pc_clouduser).

b) Click Add.

15. Click located in the upper left-hand corner of the web page.

16. Click Identity > Policies.

17. Click Create Policy.

a) For Name, enter a name for the policy.

b) For Policy Versioning, enable Keep Policy Current.c) For Policy Statements, enter: ALLOW GROUP Administrators to manage all-resources IN

TENANCY

d) Click Create.

9.8.2 Generating an SSH Public for the Oracle Cloud UserUse OpenSSL to create a private and public key in a PEM format for the previously created Oracle Clouduser.

If you're using Windows, you'll need to install Git Bash for Windows and run the commands with thattool.

Use OpenSSL to generate an API signing key pair in a PEM format. Once this task is complete, youshould have the following information stored in a file:

• The RSA public key's fingerprint.

1. Generate a private key by executing the following command:openssl genrsa -out oracle_private_key.pem 2048

Note: It is recommended to change the permissions on this file so that only you haveread/write access.

The private key is used later to create a cloud account in in Control.

2. Generate the public key by executing the following command:openssl rsa -pubout -in oracle_private_key.pem -out oracle_public_key.pem

3. Login to the Oracle Cloud Infrastructure console.

4. Click located in the upper left-hand corner of the web page.

5. Click Identity > Users.

6. Click the name of the previously created user (e.g., pc_clouduser).

7. Click Add Public Key.

8. Copy and paste the contents of the public RSA key file.

9. Click Add.A fingerprint is generated.

Proprietary Information of Altair Engineering

Page 172: Control Administrator's Guide - Altair PBS Works · Altair Control 2019.1 Administrator's Guide iv • Altair Embed SE™ ©1989-2019 (formerly solidThinking Embed® SE) • Altair

Control Administrator's Guide9 Cloud Bursting p.167

Figure 37: Public Key Fingerprint

10. Copy the fingerprint.

11. Store the fingerprint to a file.

The fingerprint is used later to create a cloud account in Control.

9.8.3 Obtain the Root Compartment IdentifierObtain the Oracle Cloud identifier of the root compartment so that you can later create a cloud accountin Control.

You must already have signed up for an Oracle Cloud account and have an associated tenancy.

Oracle documentation can be found at Understand Compartments.

When your tenancy is provisioned, a root compartment is created for you. Compartments can be usedto organize and isolate your resources to make it easier to manage and secure access to them. Yourroot compartment holds all of your cloud resources. You can think of the root compartment like a rootfolder in a file system. The first time you sign in to the Oracle Cloud Console and select a service, youwill see your root compartment. All the resources required for cloud bursting will be contained in thisroot compartment. You will need the root compartment's resource identifier to add an Oracle cloudaccount to Control. Once this task is complete, you should have the following information stored in afile:

• The compartment's Oracle Cloud ID (OCID).

1. Login to the Oracle Cloud Infrastructure console.

2. Click located in the upper left-hand corner of the web page.

3. Click Identity > Compartments.

4. Click Show located under the name of the root compartment.The compartment's OCID is displayed.

Proprietary Information of Altair Engineering

Page 173: Control Administrator's Guide - Altair PBS Works · Altair Control 2019.1 Administrator's Guide iv • Altair Embed SE™ ©1989-2019 (formerly solidThinking Embed® SE) • Altair

Control Administrator's Guide9 Cloud Bursting p.168

5. Click Copy to copy the OCID.

6. Store the OCID to a file.

The OCID is used later to create a cloud account in Control.

9.8.4 Obtain the Tenancy IdentifierObtain the Oracle Cloud identifier of the tenancy that you can later create a cloud account in Control.

You must already have signed up for an Oracle Cloud account and have an associated tenancy.

You will need the tenancy's Oracle Cloud identifier to add an Oracle cloud account to Control. Once thistask is complete, you should have the following information stored in a file:

• The tenancy's Oracle Cloud ID (OCID).

1. Login to the Oracle Cloud Infrastructure console.

2. Click located in the upper left-hand corner of the web page.

3. Click Administration > Tenancy Details.

4. Under Tenancy Information, click Show located to the right of OCID:

Figure 38: Tenancy OCID

The tenancy's OCID is displayed.

5. Click Copy to copy the OCID.

6. Store the OCID to a file.

The OCID is used later to create a cloud account in Control.

9.8.5 Add an Oracle Cloud AccountAdd an existing Oracle Cloud account to Control so that you can create a cloud bursting scenario or anappliance.

You must have created an Oracle Cloud user and added a public key for the user.

You should have the following information stored in a file:

• User OCID generated when creating the Oracle user.

• Tenancy OCID.

Proprietary Information of Altair Engineering

Page 174: Control Administrator's Guide - Altair PBS Works · Altair Control 2019.1 Administrator's Guide iv • Altair Embed SE™ ©1989-2019 (formerly solidThinking Embed® SE) • Altair

Control Administrator's Guide9 Cloud Bursting p.169

• Root compartment OCID.

• Fingerprint of the public key generated for the Oracle user.

• RSA private key that was generated for the Oracle user.

This information is necessary to create a cloud account in Control. If you do not have this informationsee Create an Oracle Cloud User Account, Generating an SSH Public for the Oracle Cloud User, Obtainthe Root Compartment Identifier and Obtain the Tenancy Identifier.

1. Login to Control.

2. Click the Cloud tab.

3. Under Infrastructure, click Cloud.

4. Click .

5. Enter the following to add a cloud account:

a) For Account name, enter any name for the cloud account.

The name can be anything meaningful to your organization, e.g., oracle_cloudaccount.

b) For User OCID, enter the user OCID generated when creating the Oracle cloud user.

c) For Tenant OCID, enter the tenancy OCID .

d) For Compartment OCID, enter the root compartment OCID.

e) For Fingerprint, enter the fingerprint generated when adding the public SSH key for theOracle user.

f) For Private Key, enter the generated RSA private key.

6. Click Create account.

9.8.6 Create a Virtual Cloud NetworkCreate a Virtual Cloud Network (VCN) so that you can perform cloud bursting.

You must already have signed up for an Oracle Cloud account and have an associated tenancy.

Oracle documentation can be found at Overview of Networking and Creating a Virtual Cloud Network.

An Oracle VCN must be created to create a bursting scenario in Control.

1. Login to the Oracle Cloud Infrastructure console.

2. Click located in the upper left-hand corner of the web page.

3. Click Networking > Virtual Cloud Networks.

4. Choose a region based on the geographical location of your users.

Proprietary Information of Altair Engineering

Page 175: Control Administrator's Guide - Altair PBS Works · Altair Control 2019.1 Administrator's Guide iv • Altair Embed SE™ ©1989-2019 (formerly solidThinking Embed® SE) • Altair

Control Administrator's Guide9 Cloud Bursting p.170

Figure 39: Select a Region

5. ClickCreate Virtual Cloud Network.

6. Enter the following to create a VNC:

a) For CREATE IN COMPARTMENT, select the root compartment.

b) For NAME, enter a name for the VNC.

The name can be anything meaningful to your organization, e.g., bursting_vnc.

c) Enable CREATE VIRTUAL CLOUD NETWORK PLUS RELATED RESOURCES.

Choosing this option automatically creates a VCN with a CIDR block 10.0.0.0/16, an internetgateway, a route rule to enable traffic to and from the internet gateway, the default securitylist, the default set of DHCP options, and one public subnet per availability domain.

7. Click Create Virtual Cloud Network.A summary of the VNC, internet gateway, default route table and subnets is displayed.

8. Click Close.A list of VNCs is displayed.

9. Click the name of the VNC.The subnets are displayed. A subnet is created for each availability domain (data center) locatedin the previously selected region.

Figure 40: Virtual Machine Subnets and Associated Availability Domain

Next, create a virtual machine that is hosted in one of the region's availability domains.

Proprietary Information of Altair Engineering

Page 176: Control Administrator's Guide - Altair PBS Works · Altair Control 2019.1 Administrator's Guide iv • Altair Embed SE™ ©1989-2019 (formerly solidThinking Embed® SE) • Altair

Control Administrator's Guide9 Cloud Bursting p.171

9.8.7 Creating a Virtual MachineCreate a CentOS 7 virtual machine (VM) in Oracle Cloud.

You must already have signed up for an Oracle Cloud account, have an associated tenancy and havecreated a VNC in a region that is geographically close to your users. The VNC should have a subnetassociated to each of the region's availability domains. If you have not created a VNC see Create aVirtual Cloud Network.

Virtual machines are hosted in availability domains (data centers) located in a region and are basedon predefined VM shapes. Before proceeding, determine the VM shape that your site requires for cloudbursting based on the number of CPUs, memory, disk space, network bandwidth, and virtual networkinterface cards. While selecting the shape for a VM, consider the nature of the applications that you planto deploy on the instance, the number of users that you expect to use the applications, and also howyou expect the load to scale in the future. Remember to also factor in the CPU and memory resourcesthat are necessary for the operating system.

Next, verify that your Oracle tenancy has the appropriate service limits for the VM shape and theregion's availability domains Check Tenancy Service Limits .

1. Login to the Oracle Cloud Infrastructure console.

2. Click located in the upper left-hand corner of the web page.

3. Click Compute > Instances.

4. Choose the region where the previously created VNC is hosted.

Figure 41: Select a Region

5. For COMPARTMENT, select the root compartment.

Proprietary Information of Altair Engineering

Page 177: Control Administrator's Guide - Altair PBS Works · Altair Control 2019.1 Administrator's Guide iv • Altair Embed SE™ ©1989-2019 (formerly solidThinking Embed® SE) • Altair

Control Administrator's Guide9 Cloud Bursting p.172

Figure 42: Choose Compartment

6. Click Create Instance.

7. Enter the following to create a virtual machine:

a) For NAME, enter a name for the VM.

The name can be anything meaningful to your organization, e.g., bursting_vm.

b) For AVAILABILITY DOMAIN, choose one of the region's availability domains.

The virtual machine is hosted in the chosen availability domain (data center). Choose theavailability domain that best suits your site's cloud bursting requirements based on themachine type of the virtual machine (VM shape) and service limits.

c) For BOOT VOLUME, enable ORACLE-PROVIDED OS IMAGE.

d) For IMAGE OPERATING SYSTEM, choose CentOS 7.

e) For SHAPE TYPE, enable VIRTUAL MACHINE.

f) For SHAPE, select a VM shape.

Choose the VM shape that best suits your site's cloud bursting requirements based onnumber of CPUs, memory, disk space, and network bandwidth.

g) For IMAGE VERSION, select the latest available one.

h) For BOOT VOLUME CONFIGURATION, enable CUSTOM BOOT VOLUME SIZE and enter aboot volume size in GBs.

i) Enable PASTE SSH KEYS and copy the SSH public key (i.e., id_rsa.pub) of a user accountthat exists on your site's PBS Server and paste it.

This user will have sudo rights and will be able to SSH into the VM to install the PBS MoM.

j) For VIRTUAL CLOUD NETWORK, choose the VCN that was created for cloud bursting.

k) For SUBNET, choose the subnet associated with the previously chosen availability domain.

8. Click Create Instance.

Creating the virtual machine may take some time. It is done when the state is "Running".

Proprietary Information of Altair Engineering

Page 178: Control Administrator's Guide - Altair PBS Works · Altair Control 2019.1 Administrator's Guide iv • Altair Embed SE™ ©1989-2019 (formerly solidThinking Embed® SE) • Altair

Control Administrator's Guide9 Cloud Bursting p.173

Figure 43: Running Virtual Machine

Check Tenancy Service LimitsCheck the service limit for your tenancy before creating the components necessary for cloud bursting.

You must already have signed up for an Oracle Cloud account and have an associated tenancy.

When you sign up for Oracle Cloud Infrastructure, a set of service limits are configured for yourtenancy. The service limit is the quota or allowance set on a resource. For example, your tenancy isallowed a maximum number of compute instances (virtual machines) per availability domain. Theselimits are generally established with your Oracle sales representative when you purchase Oracle CloudInfrastructure. Oracle documentation can be found at Service Limits and Regions and AvailabilityDomains.

When you reach the service limit for a resource, you receive an error when you try to create a newresource of that type. You cannot create a new resource until you are granted an increase to yourservice limit or you terminate an existing resource.

View your tenancy's limits to ensure that there are sufficient resources available in a region's availabilitydomains.

1. Login to the Oracle Cloud Infrastructure console.

2. Choose the region where the previously created VNC is hosted.

Proprietary Information of Altair Engineering

Page 179: Control Administrator's Guide - Altair PBS Works · Altair Control 2019.1 Administrator's Guide iv • Altair Embed SE™ ©1989-2019 (formerly solidThinking Embed® SE) • Altair

Control Administrator's Guide9 Cloud Bursting p.174

Figure 44: Select a Region

3. Click located in the upper left-hand corner of the web page.

4. Click Governance > Service Limits.

5. Scroll down to the Service Limits section.

6. Click Compute.

Availability domains (data centers) for the region are displayed. For each resource (VM shape) thenumber of nodes that can be burst in the corresponding availability domains are displayed. In thebelow example, three nodes can be burst in each data center in the us-phoenix-1 region for theVM Standard1.1 shape.

Figure 45: Virtual Machine Type Limits

7. Verify that the appropriate service limits are set for your tenancy based on the VM shape chosenfor the virtual machine and the region's availability domains.

To request an increase a service limits for your tenancy see Requesting a Service Limit Increase.

Proprietary Information of Altair Engineering

Page 180: Control Administrator's Guide - Altair PBS Works · Altair Control 2019.1 Administrator's Guide iv • Altair Embed SE™ ©1989-2019 (formerly solidThinking Embed® SE) • Altair

Control Administrator's Guide9 Cloud Bursting p.175

9.8.8 Installing and Configuring the PBS MoM on the VMInstall the PBS MoM on the virtual machine and configure the VM per your site's environment.

Log into the VM using the default user "opc" and the private SSH key of the user account providedduring the creation of the VM. For more information see Accessing an Instance from UNIX and UNIX-Like Systems.

A cloud-init script is provided during the creation of the bursting scenario. This script is ran after acloud node is burst and is used to set up the system environment so that nodes burst in the cloud cancommunicate with the PBS Professional Server, therefore installation of cloud-init on the VM is required.For more information about installing cloud-init, refer to the Cloudinit Documentation.

1. Login to your site's PBS Server as the user account provided during the creation of the VM.

2. SSH into the virtual machine using the default user "opc", the private SSH key of the user accountprovided during the creation of the VM and the External IP assigned to the VM.ssh -i PRIVATE_KEY_PATH opc@PUBLIC_IP_ADDR

Where PRIVATE_KEY_PATH is the path to the file that contains the private SSH key of the useraccount provided during the creation of the VM and PUBLIC_IP_ADDR is the public IP address ofthe VM.

Figure 46: Bursting Virtual Machine

3. Enter the command:sudo -i

4. Using the PBS Professional Installation and Upgrade Guide, install and configure the PBS MOM.

5. Once the PBS MOM is installed, edit the /etc/pbs.conf file.

6. Change the value of PBS_SERVER to the hostname of the PBS Server.

7. Save the file.

8. Edit the file PBS_HOME/mom_priv/config where PBS_HOME is the home directory of PBS.

9. Change the value of $clienthost to the hostname of the PBS Server.

Proprietary Information of Altair Engineering

Page 181: Control Administrator's Guide - Altair PBS Works · Altair Control 2019.1 Administrator's Guide iv • Altair Embed SE™ ©1989-2019 (formerly solidThinking Embed® SE) • Altair

Control Administrator's Guide9 Cloud Bursting p.176

10. Save the file.

11. Configure the VM for your site's environment such as pointing it the DNS server, mounting filesystems, connecting it to the authentication service, installing any applications, etc.

12. Install cloud-init.

13. Connect the VM to your PBS server and submit a job to ensure it is working as expected.

9.8.9 Create an OS ImageCreate an image of the virtual machine configured per your site's requirements.

You must have already create a virtual machine, installed the PBS MoM on the VM and configured theVM per your site's environment.

Oracle documentation can be found at Managing Custom Images.

1. Login to the Oracle Cloud Infrastructure console.

2. Click located in the upper left-hand corner of the web page.

3. Click Compute > Instances.A list of virtual machines is displayed.

4. Click the name of the virtual machine created for cloud bursting.

5. Click Create Custom Image.

6. Enter the following to create a custom image:

a) For CREATE IN COMPARTMENT, select the root compartment.

b) For NAME, enter a name for the image.

The name can be anything meaningful to your organization, e.g., bursting_image.

7. Click Create Custom Image.

It may take some time to create the image. Do not proceed until the image is created.

You can now delete the virtual machine so that you are no longer charged for it.

9.8.10 Create an Oracle Cloud Bursting ScenarioCreate an Oracle bursting scenario so that PBS execution nodes can be burst in the cloud.

Open a browser window and login to the Oracle Cloud Infrastructure console. Open a second browserwindow and login to Control.

A cloud-init script is provided during the creation of the bursting scenario. This script is ran after acloud node is burst and is used to set up the system environment so that nodes burst in the cloud cancommunicate with the PBS Professional Server. For more information about writing cloud-init scripts,refer to the Cloudinit Documentation and the Cloud Bursting chapter of the ControlAdministrator'sGuide.

1. Navigate to the Control browser window.

2. Click Cloud.

Proprietary Information of Altair Engineering

Page 182: Control Administrator's Guide - Altair PBS Works · Altair Control 2019.1 Administrator's Guide iv • Altair Embed SE™ ©1989-2019 (formerly solidThinking Embed® SE) • Altair

Control Administrator's Guide9 Cloud Bursting p.177

3. Under Infrastructure, click Bursting.

4. Click Add Bursting Scenario.

5. Enter information about the bursting scenario such as name, cloud provider, cloud account, andgeographic location.

a) For Name, enter a name for the cloud bursting scenario.

b) For Description, enter a description.

c) For Cloud Account, choose a previously created Oracle cloud account from the list.

d) For Select a region, select the availability domain where the cloud bursting virtual machine ishosted.

Figure 47: Virtual Machine Availability Domain

6. Click Next.Enter information about the Oracle resources necessary to burst in the cloud.

7. For Domain name, provide the domain name of your organization for the cloud bursting network.altair.com, burst.altair.com

8. For Hostname prefix, enter a prefix for the names of the burst nodes.If "node" is entered , the cloud bursting nodes will be available to PBS Professional as node1,node2, node3, etc.

9. Navigate to the Oracle Cloud Infrastructure browser window.

10. Click located in the upper left-hand corner of the web page.

11. Click Networking > Virtual Cloud Networks.

12. Click the name of the VCN create for cloud bursting.

13. Locate the subnet associated with the availability domain where the cloud bursting virtual machineis hosted.

Proprietary Information of Altair Engineering

Page 183: Control Administrator's Guide - Altair PBS Works · Altair Control 2019.1 Administrator's Guide iv • Altair Embed SE™ ©1989-2019 (formerly solidThinking Embed® SE) • Altair

Control Administrator's Guide9 Cloud Bursting p.178

Figure 48: Subnet and Associated Availability Domain

14. Click Show located under the name of the subnet.The subnet's OCID is displayed.

15. Click Copy to copy the OCID.

16. Navigate to the Control browser window.

17. For Subnet OCID, paste the subnet's OCID copied from Oracle.

18. Enable the Add public IP to VMs if you need to connect to the cloud bursting nodes from theinternet.

It is recommended to leave this disabled.

19. Navigate to the Oracle Cloud Infrastructure browser window.

20. Click located in the upper left-hand corner of the web page.

21. Click Compute > Custom Images.A list of custom images is displayed.

22. Locate the custom image created from the cloud bursting virtual machine.

23. Click the Show link below the name of the image to view the OCID.

24. Click Copy to copy the OCID of the image.

25. Navigate to the Control browser window.

26. For OS Image URI, paste the link copied from Oracle.

27. For Cloud-init script, click Browse, browse for the script file and add it.

This script will run after a cloud node is burst.

28. Add public SSH keys for any users that you want to have access to the burst nodes:

a) Login to the PBS Server.

b) Copy the public SSH key for the user.

Public key files are usually stored in /.ssh in the user's home directory.

c) Navigate to the Control browser window.

d) Click Add to add the public key.

e) For Public SSH keys, paste the public SSH key.

Proprietary Information of Altair Engineering

Page 184: Control Administrator's Guide - Altair PBS Works · Altair Control 2019.1 Administrator's Guide iv • Altair Embed SE™ ©1989-2019 (formerly solidThinking Embed® SE) • Altair

Control Administrator's Guide9 Cloud Bursting p.179

f) Repeat steps a through e to add additional public SSH keys.

The public keys are copied to each burst node for secure connectivity.

29. Click Instantiate scenario.

Proprietary Information of Altair Engineering

Page 185: Control Administrator's Guide - Altair PBS Works · Altair Control 2019.1 Administrator's Guide iv • Altair Embed SE™ ©1989-2019 (formerly solidThinking Embed® SE) • Altair

Control Administrator's Guide9 Cloud Bursting p.180

9.9 Configure Orange Cloud Flexible Engine forCloud BurstingCreate the necessary components in Orange Cloud Flexible Engine that are required for cloud burstingand then create a cloud bursting scenario.

Purchase an Orange Business Services account and use your Orange ID and password to access theFlexible Engine console. For more information visit Orange Cloud.

9.9.1 Create an Orange Cloud Flexible Engine UserAccountCreate an Orange Cloud Flexible Engine user account so that you can later create a cloud account inControl.

You must already have signed up for an Orange Cloud account. You should be able to login to theOrange Cloud Customer space (https://selfcare.cloud.orange-business.com/) with the credentialsprovided to you with your Orange Cloud account. You will also be provided with a Domain name whenyou sign up for your Orange Cloud account.

An Orange Cloud user account must be created to add an Orange Cloud Flexible Engine account toControl. Once this task is complete, you should have the following information stored in a file:

• Username and password

• API password

This information is necessary to create a cloud account in Control.

1. Go to the Orange Cloud Customer space login page.

2. Enter your Orange Cloud credentials.

3. Click Your services.

Figure 49: Orange Cloud Customer Space Services

4.In the navigation bar on the top click Users

5. Click Add user.6. Enter the following user details:

Proprietary Information of Altair Engineering

Page 186: Control Administrator's Guide - Altair PBS Works · Altair Control 2019.1 Administrator's Guide iv • Altair Embed SE™ ©1989-2019 (formerly solidThinking Embed® SE) • Altair

Control Administrator's Guide9 Cloud Bursting p.181

a) For Civility, choose Mrs or Mr

b) For Last name, enter the user's last name.

c) For First name, enter the user's first name.

d) For Login, enter a login name for the user.

e) For Email, enter the user's email address.

f) For Phone number, enter the user's phone number.

g) For Mobile phone, enter the user's mobile phone number.

h) For Preferred language, choose the language in which the application should be displayed.

i) Click next.7. In the Roles section enter these details

a) For Billing, choose Visitor.b) For Contracts, choose Account Manager.c) For Dashboard, choose Visitor.d) For Documents, choose Visitor.e) For Orders, choose Visitor.f) For Services, choose Visitor.g) For Subscriptions, choose Visitor.h) For Support, choose Visitor.i) For Users. choose Manager.j) For Flexible Engine Console. choose admin.

k) Click next.8. In the Summary section review your choices.

Click previous to edit your choices entries.

9. Click finish.The new user account is created and displayed in the list of users. Emails are sent to the emailaddress you specified. The emails will contain:

1. Orange ID (Domain Name).

2. Link to set Orange Password.

3. Link to access Cloud Customer Space.

4. Link to log in to the Flexible Engine Console.

5. Link to define your API password.

10. Click the link in the email to set your Orange Password.

11. Click the link in the email to set your API Password.

12. Store the API Password to a file.

The API Password is used later to create a cloud account in Control.

Proprietary Information of Altair Engineering

Page 187: Control Administrator's Guide - Altair PBS Works · Altair Control 2019.1 Administrator's Guide iv • Altair Embed SE™ ©1989-2019 (formerly solidThinking Embed® SE) • Altair

Control Administrator's Guide9 Cloud Bursting p.182

9.9.2 Select a RegionDefine a region in the Orange Cloud Flexible Engine console to set up the infrastructure for cloudbursting.

A region is a geographic area where resources used by your ECSs are located. ECSs in the same regioncan communicate with each other over an intranet, but ECSs in different regions cannot. Before settingup the infrastructure for cloud bursting, it is important to ensure that all the resources are defined inthe same region. An Authorization URL is required for adding the Orange Cloud Flexible Engine cloudaccount in Control. This is based on the region selected.

1. Login to the Orange Cloud Flexible Engine console.

2. In the navigation bar on the top select the region for setting up your infrastructure.

Figure 50: Select a Region

Note: For Auth (IAM) URL (Authorization URL), enter the url in the following formatbased on the region you chose in the Orange Cloud Flexible Engine console.

https://iam.<orange region>.<console link> e.g. https://iam.eu-west-0.prod-cloud-ocb.orange-business.com

3. Store the region and Auth (IAM) URL in a file. You will use this to register the account in Control.

9.9.3 Add an Orange Cloud Flexible Engine AccountAdd an existing Orange Cloud Flexible Engine account to Control so that you can create a cloud burstingscenario or an appliance.

You must have:

• Created an Orange Cloud Flexible Engine user account.

• Obtained the Domain name for the Orange Cloud Flexible Engine account.

• Obtained the API password for the user.

• Defined the region where you will deploy your cloud infrastructure.

• Obtained the Auth (IAM) URL based on the selected region.

This information is necessary to create a cloud account in Control. If you do not have this informationsee Create an Orange Cloud Flexible Engine User Account and Select a Region.

1. Login to Control.

Proprietary Information of Altair Engineering

Page 188: Control Administrator's Guide - Altair PBS Works · Altair Control 2019.1 Administrator's Guide iv • Altair Embed SE™ ©1989-2019 (formerly solidThinking Embed® SE) • Altair

Control Administrator's Guide9 Cloud Bursting p.183

2. Click the Cloud tab.

3. Under Infrastructure, click Cloud.

4. Click .

5. Enter the following to add a cloud account:

a) For Account name, enter any name for the cloud account.

The name can be anything meaningful to your organization, e.g., orange_cloudaccount.

b) For Auth (IAM) URL, enter the url in the following format based on the region you chose inthe Orange Cloud Flexible Engine console.https://iam.<orange region>.<console link> e.g. https://iam.eu-west-0.prod-cloud-ocb.orange-business.com

c) For Domain name, enter the Domain Name you used to login to your Orange account. It isthe same as your Customer ID.

d) For User name, enter the username of the user account you created in the Orange CloudCustomer space.

e) For Password, enter the API password you generated after creating the user.

6. Click Create Account.

9.9.4 Check Orange Cloud Flexible Engine Account ServiceQuotaView the your Orange Cloud Flexible Engine resource usage and limits from the console.

Quotas are used to limit the number of resources available to users. It is important to ensure you arenot exceeding your quota while setting up the resources for cloud bursting. If the existing resourcequota cannot meet your service requirements, you can submit a work order to increase your quota.Once your application is approved, Orange Cloud Flexible Engine will update your resource quotaaccordingly and send you a notification.

1. Login to the Orange Cloud Flexible Engine console.

2. In the navigation bar on the top right hand side, click .

Figure 51: Check Quotas for Resources

Information about resources usage and availability is displayed.

Proprietary Information of Altair Engineering

Page 189: Control Administrator's Guide - Altair PBS Works · Altair Control 2019.1 Administrator's Guide iv • Altair Embed SE™ ©1989-2019 (formerly solidThinking Embed® SE) • Altair

Control Administrator's Guide9 Cloud Bursting p.184

Figure 52: Resources Quotas Table

9.9.5 Create a Virtual Private CloudCreate a Virtual Private Cloud (VPC) so that you can perform cloud bursting.

Orange Cloud Flexible Engine documentation for VPC can be found at:Virtual Private CloudDocumentation

1. Login to the Orange Cloud Flexible Engine console.

2. In the top navigation bar select the region where you wish to deploy your cloud infrastructure.

3. From the Network section click Virtual Private Cloud.

4. Click + Create VPC.

5. In the Basic Information section:

a) For Region, ensure the VPC is the same region as the other resources.

b) For Name, enter a name for the VPC.

c) For CIDR Block, enter CIDR values for the VPC.

6. In the Subnet Settings section:

a) For the Subnet Settings choose the AZ (Availability Zone) as the same as the region.

b) For Subnet Name, enter a name to match the VPC Name.

c) Enter CIDR Block for Subnet.

d) For Advanced Settings, click Default.7. Review the Configuration information.

8. Click Create Now.

9. Once the VPC is created, click the Back to VPC List.10. Click Security Group in the left hand side menu.

11. Click + Create Security Group.

12. For Name, enter a name for the Security Group.

13. For Description, enter a suitable Description.

Proprietary Information of Altair Engineering

Page 190: Control Administrator's Guide - Altair PBS Works · Altair Control 2019.1 Administrator's Guide iv • Altair Embed SE™ ©1989-2019 (formerly solidThinking Embed® SE) • Altair

Control Administrator's Guide9 Cloud Bursting p.185

14. Click OK

Note: By default, the Outbound and Inbound traffic over IPv4 is open. You can addfirewall rules to this security group if required.

9.9.6 Creating a Virtual MachineCreate a Elastic Cloud Server (ECS) using a public image provided by Orange Cloud Flexible Engine forcloud bursting.

You must have already created a Virtual Private Network with a subnet and assigned a Security Group.If you do not have this information, see Create a Virtual Private Cloud

1. Login to the Orange Cloud Flexible Engine console.

2. In the Computing section and click Elastic Cloud Server.3. From the menu on the left hand side click Key Pair.4. Click + Create Key Pair.5. Enter a Name for the Key Pair.

6. Click OK.

7. Save the Key Pair (.pem) file to your local disk in a secure location.

The information in this .pem file is used later to SSH into the VM.

8. Click OK to confirm that you have downloaded the Key Pair file.

9. From the menu on the left hand side, click Elastic Cloud Server.10. Click + Create ECS.

11. For Region, click the region you selected for setting up the infrastructure.

12. For AZ (Availability Zone), select the AZ related to the region.

13. In the Specifications section:

a) For ECS type, click one of the flavor names.

Note: Orange Cloud Flexible Engine provides a set of predefined ECS typesfor specific requirements. Click a flavor name to get the list of availableconfigurations.

b) Review the specifications you have selected.

14. In the Image section:

a) Click Public image.

b) From the drop down menu select CentOS

c) From the version drop down menu select Select OBS_U_CentOS_7.2(40GB)

15. In the Disk section select the defaults.

16. In the VPC section:

a) For VPC, select the VPC you created from the drop down menu.

b) For NIC, choose the default primary NIC.

Proprietary Information of Altair Engineering

Page 191: Control Administrator's Guide - Altair PBS Works · Altair Control 2019.1 Administrator's Guide iv • Altair Embed SE™ ©1989-2019 (formerly solidThinking Embed® SE) • Altair

Control Administrator's Guide9 Cloud Bursting p.186

c) For Security Group, select the Security Group you created for the VPC.

d) For EIP, click Automatically assigne) For Bandwidth, specify it as 5 Mbit/s.

17. For Login Mode, select the Key Pair you generated earlier from the drop down menu.

18. For Auto Recovery, click Enable.

19. For Advanced Settings, click Do not configure.

20. For ECS Name, enter a name.

21. For Quantity, specify 1.

22. Review the Current Configuration.

23. Click Create Now.

24. Review the Specifications.

25. Click Submit.The ECS (Virtual Machine) is created and displayed in the list of ECS.

9.9.7 Installing and Configuring the PBS MoM on the VMInstall the PBS MoM on the virtual machine and configure the VM per your site's environment.

Log into the VM using the default user "cloud" and the .pem generated during the creation of the VM.For more information about logging into the Linux ECS, refer to the Elastic Cloud Server User Guide.

A cloud-init script is provided during the creation of the bursting scenario. This script is ran after acloud node is burst and is used to set up the system environment so that nodes burst in the cloud cancommunicate with the PBS Professional Server, therefore installation of cloud-init on the VM is required.For more information about installing cloud-init, refer to the Cloudinit Documentation.

1. Login to your site's PBS Server.

2. Login to the Flexible Engine console.

3. In the Computing section, click Elastic Cloud Server.4. In the search box above the upper right corner of the ECS list, enter the ECS name, IP address, or

ID, and click .

5. Click the name of the target ECS.

6. The page providing details about the ECS is displayed.

7. Copy the Public IP address (External IP) of the ECS.

8. SSH into the VM using the default user "cloud", the .pem file you generated when creating the VMand the External IP assigned to the VM.ssh -i /path/my-key-pair.pem cloud@IPV4PublicIP

where /path/my-key-pair.pem is the path to the .pem file downloaded while creating the virtualmachine and IPV4PublicIP is the public IP address of the virtual machine.

9. Enter the command:sudo -i

10. Using the PBS Professional Installation and Upgrade Guide, install and configure the PBS MOM.

Proprietary Information of Altair Engineering

Page 192: Control Administrator's Guide - Altair PBS Works · Altair Control 2019.1 Administrator's Guide iv • Altair Embed SE™ ©1989-2019 (formerly solidThinking Embed® SE) • Altair

Control Administrator's Guide9 Cloud Bursting p.187

11. Once the PBS MOM is installed, edit the /etc/pbs.conf file.

12. Change the value of PBS_SERVER to the hostname of the PBS Server.

13. Save the file.

14. Edit the file PBS_HOME/mom_priv/config.

where PBS_HOME is the home directory of PBS.

15. Change the value of $clienthost to the hostname of the PBS Server.

16. Save the file.

17. Configure the VM for your site's environment such as pointing it the DNS server, mounting filesystems, connecting it to the authentication service, installing any applications, etc.

18. Install cloud-init.

19. Connect the VM to your PBS server and submit a job to ensure it is working as expected.

9.9.8 Create an OS ImageCreate an image of the virtual machine configured per your site's requirements.

You must have already created a ECS (VM), installed the PBS MoM on the VM and configured it per yoursite's environment.

Orange Cloud Flexible Engine documentation can be found at Creating a Linux Private Image Using anECS. Before creating the Image from the ECS, you must have.

• A Linux ECS in the Stopped state.

• Configured DHCP for the NICs of the ECS

• Configured Network attributes of the ECS

• Detached Data Disks from the ECS

1. Login to the Flexible Engine console.

2. In the Computing section, click Image Management Service.

3. On the Image Management Service page, click + Create Private Image.

4. In the Image Type and Source section, .

a) For Type, click System disk imageb) For Source, click ECSc) Select the target ECS from the ECS list.

5. Set the required information, such as Name and Description.

6. Click Create Now.

7. Confirm the parameters and click Submit.8. Switch back to the Image Management Service page to view the image status.

The time required for creating an image varies depending on the image file size. Generally, ittakes about 20 minutes to create an image. The image is successfully created when its imagestatus changes to Normal.

Proprietary Information of Altair Engineering

Page 193: Control Administrator's Guide - Altair PBS Works · Altair Control 2019.1 Administrator's Guide iv • Altair Embed SE™ ©1989-2019 (formerly solidThinking Embed® SE) • Altair

Control Administrator's Guide9 Cloud Bursting p.188

Note: Do not perform any operation on the selected ECS or its associated resourcesduring image creation.

You can now delete the virtual machine so that you are no longer charged for it.

9.9.9 Create a Orange Cloud Cloud Bursting ScenarioCreate an Orange Cloud Flexible Engine bursting scenario so that PBS execution nodes can be burst inthe cloud.

Open a browser window and login to the Orange Cloud Flexible Engine console. Open a second browserwindow and login to Control.

A cloud-init script is provided during the creation of the bursting scenario. This script is ran after acloud node is burst and is used to set up the system environment so that nodes burst in the cloud cancommunicate with the PBS Professional Server. For more information about writing cloud-init scripts,refer to the Cloudinit Documentation and the Cloud Bursting chapter of the ControlAdministrator'sGuide.

1. Navigate to the Control browser window.

2. Click Cloud.

3. Under Infrastructure, click Bursting.

4. Click Add Bursting Scenario.

5. Enter information about the bursting scenario such as name, cloud provider, cloud account, andgeographic location.

a) For Name, enter a name for the cloud bursting scenario.

b) For Description, enter a description.

c) For Cloud Account, choose a previously created Orange Cloud Flexible Engine account fromthe list.

d) For Select a region, select the availability domain where the cloud bursting virtual machine ishosted.

Figure 53: Virtual Machine Availability Domain

6. Click Next.7. For Domain name, provide the domain name of your organization for the cloud bursting network.

Proprietary Information of Altair Engineering

Page 194: Control Administrator's Guide - Altair PBS Works · Altair Control 2019.1 Administrator's Guide iv • Altair Embed SE™ ©1989-2019 (formerly solidThinking Embed® SE) • Altair

Control Administrator's Guide9 Cloud Bursting p.189

altair.com, burst.altair.com

8. For Hostname prefix, enter a prefix for the names of the burst nodes.If "node" is entered , the cloud bursting nodes will be available to PBS Professional as node1,node2, node3, etc.

9. Navigate to the Orange Cloud Flexible Engine console browser window.

10. Click Service List.

Figure 54: Orange Cloud Flexible Engine Console

11. Under Network, click Virtual Private Cloud.

12. Click Virtual Private Cloud from the menu located on the left-hand side of the web page.

13. Click the name of the VPC you created for cloud bursting.

14. Click the name of the Subnet for the VPC.

15. Copy the Subnet ID.

16. Navigate to the Control browser window.

17. For Subnet ID, paste the subnet's ID copied from the Orange Cloud Flexible Engine console.

18. In the Orange Cloud Flexible Engine console browser window click Security Groupfrom the lefthand side menu.

19. Click the name of the Security Group you created for the VPC.

20. Copy the ID of the Security Group.

21. Navigate to the Control browser window.

22. For Security Group ID, paste the security group's ID you copied from the Orange Cloud FlexibleEngine console.

23. Enable the Add public IP to VMs if you need to connect to the cloud bursting nodes from theinternet.

It is recommended to leave this disabled.

24. Navigate to the Orange Cloud Flexible Engine console browser window.

25. Click Service List.

Figure 55: Orange Cloud Flexible Engine Console

26. Under Computing, click Image Management Service.

27. Click the Private Images tab.

28. Click the name of the VM image you created for cloud bursting.

29. Copy the ID of the image.

30. Navigate to the Control browser window.

31. For OS Image URI, paste the image ID you copied from the Orange Cloud Flexible Engine console.

Proprietary Information of Altair Engineering

Page 195: Control Administrator's Guide - Altair PBS Works · Altair Control 2019.1 Administrator's Guide iv • Altair Embed SE™ ©1989-2019 (formerly solidThinking Embed® SE) • Altair

Control Administrator's Guide9 Cloud Bursting p.190

32. For Cloud-init script, click Browse, browse for the script file and add it.

This script will run after a cloud node is burst.

33. Add public SSH keys for any users that you want to have access to the burst nodes:

a) Login to the PBS Server.

b) Copy the public SSH key for the user.

Public key files are usually stored in /.ssh in the user's home directory.

c) Navigate to the Control browser window.

d) Click Add to add the public key.

e) For Public SSH keys, paste the public SSH key.

f) Repeat steps a through e to add additional public SSH keys.

The public keys are copied to each burst node for secure connectivity.

34. Click Instantiate scenario.

Proprietary Information of Altair Engineering

Page 196: Control Administrator's Guide - Altair PBS Works · Altair Control 2019.1 Administrator's Guide iv • Altair Embed SE™ ©1989-2019 (formerly solidThinking Embed® SE) • Altair

Control Administrator's Guide9 Cloud Bursting p.191

9.10 Configure HUAWEI Cloud for Cloud BurstingCreate the necessary components in HUAWEI Cloud that are required for cloud bursting and then createa cloud bursting scenario.

Before you can begin creating the components necessary for cloud bursting, you must create andactivate an HUAWEI Cloud account.

9.10.1 Obtain the HUAWEI Cloud AdministratorCredentialsGet the credentials for the HUAWEI Cloud administrative user account so that you can later use thisinformation to create a cloud account in Control.

You must already have signed up for an HUAWEI Cloud account.

When subscribing to HUAWEI Cloud, an administrative user account that has permissions for all systemoperations is automatically created. Use this user account to add an HUAWEI Cloud account to Control.Reset the password for this account when the password is unknown.

Note: Alternately, you can create a new user and give the user administrative privileges bysetting its User Group to "admin".

1. Login to the HUAWEI Cloud Console.

2. Click Service List.

Figure 56: HUAWEI Cloud Console

3. Under Management & Deployment, click Identity and Access Management.4. Click Users from the menu located on the left-hand side of the web page.

A list of users is displayed.

5. Click located next to a username to display the user's details.The user account listed as an "admin" is the account to use to create the cloud account in Control.

Proprietary Information of Altair Engineering

Page 197: Control Administrator's Guide - Altair PBS Works · Altair Control 2019.1 Administrator's Guide iv • Altair Embed SE™ ©1989-2019 (formerly solidThinking Embed® SE) • Altair

Control Administrator's Guide9 Cloud Bursting p.192

Figure 57: User Details

6. If you do not know the password for the administrative user account, click Set Credentials.

Figure 58: Set Password

a) Enable Set manually.

b) For Password, enter a password for the user account.

c) For Confirm Password, enter the password a second time.

d) Click OK.

You may have to confirm the password change either by email or by a SMS text.

9.10.2 Add an HUAWEI Cloud AccountAdd an existing HUAWEI Cloud account to Control so that you can create a cloud bursting scenario.

1. Login to Control.

2. Click the Cloud tab.

3. Under Infrastructure, click Cloud.

4. Click .

5. Enter the following to add a cloud account:

a) For Account name, enter any name for the cloud account.

The name can be anything meaningful to your organization, e.g., huawei_cloudaccount.

b) For Auth (IAM) URL, enter https://iam.ap-southeast-1.myhwclouds.com.

c) For Domain name, enter the Domain Name provided when your subscription to HUAWEICloud was created.

If you do not know your Domain name, contact HUAWEI Cloud support.

d) For Username, enter a username that is an HUAWEI Cloud administrator.

e) For Password, enter the password for the HUAWEI Cloud administrator.

Proprietary Information of Altair Engineering

Page 198: Control Administrator's Guide - Altair PBS Works · Altair Control 2019.1 Administrator's Guide iv • Altair Embed SE™ ©1989-2019 (formerly solidThinking Embed® SE) • Altair

Control Administrator's Guide9 Cloud Bursting p.193

6. Click Create Account.

9.10.3 Check HUAWEI Cloud Account Service QuotasView your OTC account resource usage and limits.

Quotas are used to limit the number of resources available to users. It is important to ensure you arenot exceeding your quota while setting up the resources for cloud bursting. If the existing resourcequota cannot meet your service requirements, you can submit a work order to increase your quota.Once your application is approved, HUAWEI Cloud will update your resource quota accordingly and sendyou a notification.

1. Login to the HUAWEI Cloud console.

2. Click Resources > My Quota.

Figure 59: View Quotas

Information about resources usage and availability is displayed.

Figure 60: Resources Quotas

9.10.4 Create a Virtual Private CloudCreate a Virtual Private Cloud (VPC) so that you can perform cloud bursting.

HUAWEI Cloud documentation for creating a VPC can be found at: Creating a VPC and Regions and AZs.

1. Login to the HUAWEI Cloud Console.

Proprietary Information of Altair Engineering

Page 199: Control Administrator's Guide - Altair PBS Works · Altair Control 2019.1 Administrator's Guide iv • Altair Embed SE™ ©1989-2019 (formerly solidThinking Embed® SE) • Altair

Control Administrator's Guide9 Cloud Bursting p.194

2. Click Service List.

Figure 61: HUAWEI Cloud Console

3. Under Network, click Virtual Private Cloud.

4. Click + Create VPC.

5. In the Basic Information section:

a) For Region, select a region.

Regions are geographic areas isolated from each other. Resources are region-specific andcannot be used across regions through internal network connections. For low network latencyand quick resource access, select the nearest region.

b) For Name, enter a name for the VPC.

c) For CIDR Block, enter an address range for the network using CIDR notation.

6. In the Subnet Settings section:

a) For the Subnet Settings choose the AZ (Availability Zone).

An Availability Zone is a physical location where resources use independent power suppliesand networks. AZs are physically isolated and AZs in the same VPC are interconnectedthrough an internal network.

b) For Subnet Name, enter a name for the subnet.

c) For CIDR, enter an address range for the subnet using CIDR notation.

d) For Advanced Settings, click Default.7. Click Create Now.

8. Click Back to VPC List.9. Click Security Group in the left hand side menu.

10. Click + Create Security Group.

11. For Name, enter a name for the security group.

12. For Description, enter a suitable description.

13. Click OK

Note: By default, the Outbound and Inbound traffic over IPv4 is open. You can addfirewall rules to this security group if required.

9.10.5 Creating a Virtual MachineCreate a Elastic Cloud Server (ECS) using a public image provided by HUAWEI Cloud for cloud bursting.

You must have already created a Virtual Private Network with a subnet and assigned a Security Group.If you do not have this information, see Create a Virtual Private Cloud.

HUAWEI Cloud documentation for creating an ECS (virtual machine) can be found at Purchase an ECS.

Proprietary Information of Altair Engineering

Page 200: Control Administrator's Guide - Altair PBS Works · Altair Control 2019.1 Administrator's Guide iv • Altair Embed SE™ ©1989-2019 (formerly solidThinking Embed® SE) • Altair

Control Administrator's Guide9 Cloud Bursting p.195

1. Login to the HUAWEI Cloud Console.

2. Click Service List.

Figure 62: HUAWEI Cloud Console

3. Under Computing, click Elastic Cloud Server.4. Click Key Pair from the menu located on the left-hand side of the web page.

5. Click + Create Key Pair.6. For Name, enter a name for the key pair.

7. Click OK.

8. Save the key pair (.pem) file to your local disk in a secure location.

The information in this .pem file is used later to SSH into the VM.

9. Click OK to confirm that you have downloaded the key pair file.

10. Click Elastic Cloud Server from the menu located on the left-hand side of the web page.

11. Click Buy ECS.

12. For Billing Mode, click Pay-per-use.

13. For Region, select the same region that was chosen for the previously created VPC.

14. For AZ (Availability Zone), select the same AZ that was chosen for the previously created VPC.

15. In the Type section:

a) Choose a ECS type category:

• General computing

• General computing-plus

• Memory-optimized

• Large-memory

• High-performance computing

• Disk-intensive

b) For ECS type, click one of the flavor based on the needs of your site.

16. In the Image section:

a) Click Public image.

b) For Select an OS, select CentOS.

c) For Select an OS version, select CentOS 7.2 64bit(40GB).

17. In the Disk section, select your system disk requirements.

18. In the VPC section:

a) For VPC, select the VPC you created for cloud bursting.The NIC information is automatically populated.

b) For Security Group, select the security group you created for cloud bursting.

c) For EIP, click Automatically assignd) For Bandwidth, specify it as 5 Mbit/s.

Proprietary Information of Altair Engineering

Page 201: Control Administrator's Guide - Altair PBS Works · Altair Control 2019.1 Administrator's Guide iv • Altair Embed SE™ ©1989-2019 (formerly solidThinking Embed® SE) • Altair

Control Administrator's Guide9 Cloud Bursting p.196

19. For Login Mode, select Key Pair.20. For Key Pair, select the key pair file you generated earlier.

21. For Advanced Settings, click Not required.

22. For ECS Name, enter a name for the virtual machine.

23. For Quantity, specify 1.

24. Click Next.25. Review the specifications.

26. Enable the I have read and agree to the Huawei Image Disclaimer checkbox.

27. Click Submit Application.

28. Click Back to ECS List.It may take some time to create the virtual machine. Once the ECS is created it is displayed in theECS list.

9.10.6 Installing and Configuring the PBS MoM on the VMInstall the PBS MoM on the virtual machine and configure the VM per your site's environment.

Log into the VM using the default user "root" and the .pem generated during the creation of the VM.

A cloud-init script is provided during the creation of the bursting scenario. This script is ran after acloud node is burst and is used to set up the system environment so that nodes burst in the cloud cancommunicate with the PBS Professional Server, therefore installation of cloud-init on the VM is required.For more information about installing cloud-init, refer to the Cloudinit Documentation and InstallingCloud-Init.

1. Login to the HUAWEI Cloud console.

2. Click Service List.

Figure 63: HUAWEI Cloud Console

3. Under Computing, click Elastic Cloud Server.4. Copy the Public IP address (External IP) of the ECS.

Figure 64: VM IP Address

5. Login to your site's PBS Server.

6. SSH into the VM using the default user "root", the .pem file you generated when creating the VMand the External IP assigned to the VM.

Proprietary Information of Altair Engineering

Page 202: Control Administrator's Guide - Altair PBS Works · Altair Control 2019.1 Administrator's Guide iv • Altair Embed SE™ ©1989-2019 (formerly solidThinking Embed® SE) • Altair

Control Administrator's Guide9 Cloud Bursting p.197

ssh -i /path/my-key-pair.pem root@IPV4PublicIP

where /path/my-key-pair.pem is the path to the .pem file downloaded while creating the virtualmachine and IPV4PublicIP is the public IP address of the virtual machine.

7. Using the PBS Professional Installation and Upgrade Guide, install and configure the PBS MOM.

8. Once the PBS MOM is installed, edit the /etc/pbs.conf file.

9. Change the value of PBS_SERVER to the hostname of the PBS Server.

10. Save the file.

11. Edit the file PBS_HOME/mom_priv/config.

where PBS_HOME is the home directory of PBS.

12. Change the value of $clienthost to the hostname of the PBS Server.

13. Save the file.

14. Configure the VM for your site's environment such as pointing it the DNS server, mounting filesystems, connecting it to the authentication service, installing any applications, etc.

15. Install cloud-init.

16. Connect the VM to your PBS server and submit a job to ensure it is working as expected.

9.10.7 Create an OS ImageCreate an image of the virtual machine configured per your site's requirements.

You must have already created a ECS (virtual machine), installed the PBS MoM on the ECS andconfigured it per your site's environment.

HUAWEI Cloud documentation can be found at Creating a Linux Private Image.

1. Login to the HUAWEI Cloud console.

2. Click Service List.

Figure 65: HUAWEI Cloud Console

3. Under Computing, click Image Management Service.

4. Click + Create Image.

5. For Region, select the same region that was chosen for the previously created VPC and ECS.

6. In the Image Type and Source section, .

a) For Type, click System disk image.

b) For Source, click ECS.

c) Select the virtual machine created for cloud bursting.

d) For Name, enter a name for the virtual machine.

7. Click Next.8. Review the specifications.

Proprietary Information of Altair Engineering

Page 203: Control Administrator's Guide - Altair PBS Works · Altair Control 2019.1 Administrator's Guide iv • Altair Embed SE™ ©1989-2019 (formerly solidThinking Embed® SE) • Altair

Control Administrator's Guide9 Cloud Bursting p.198

9. Enable the I have read and agree to the Statement of Commitment to Image Creationand Huawei Image Disclaimer checkbox.

10. Click Submit.11. Click Back to Image List.

The time required for creating an image varies depending on the image file size. Generally, ittakes about 20 minutes to create an image. The image is successfully created when its imagestatus changes to Normal.

Note: Do not perform any operation on the selected ECS or its associated resourcesduring image creation.

You can now delete the virtual machine so that you are no longer charged for it.

9.10.8 Create an HUAWEI Cloud Bursting ScenarioCreate an HUAWEI Cloud bursting scenario so that PBS execution nodes can be burst in the cloud.

Open a browser window and login to the HUAWEI Cloud console. Open a second browser window andlogin to Control.

A cloud-init script is provided during the creation of the bursting scenario. This script is ran after acloud node is burst and is used to set up the system environment so that nodes burst in the cloud cancommunicate with the PBS Professional Server. For more information about writing cloud-init scripts,refer to the Cloudinit Documentation and the Cloud Bursting chapter of the ControlAdministrator'sGuide.

1. Navigate to the Control browser window.

2. Click Cloud.

3. Under Infrastructure, click Bursting.

4. Click Add Bursting Scenario.

5. Enter information about the bursting scenario such as name, cloud provider, cloud account, andgeographic location.

a) For Name, enter a name for the cloud bursting scenario.

b) For Description, enter a description.

c) For Cloud Account, choose a previously created HUAWEI Cloud account from the list.

d) For Select a region, select the same region that was chosen for the previously created VMand VPC.

6. Click Next.Enter information about the HUAWEI Cloud resources necessary to burst in the cloud.

7. For Domain name, provide the domain name of your organization for the cloud bursting network.altair.com, burst.altair.com

8. For Hostname prefix, enter a prefix for the names of the burst nodes.If "node" is entered , the cloud bursting nodes will be available to PBS Professional as node1,node2, node3, etc.

Proprietary Information of Altair Engineering

Page 204: Control Administrator's Guide - Altair PBS Works · Altair Control 2019.1 Administrator's Guide iv • Altair Embed SE™ ©1989-2019 (formerly solidThinking Embed® SE) • Altair

Control Administrator's Guide9 Cloud Bursting p.199

9. Navigate to the HUAWEI Cloud console browser window.

10. Click Service List.

Figure 66: HUAWEI Cloud Console

11. Under Network, click Virtual Private Cloud.

12. Click Virtual Private Cloud from the menu located on the left-hand side of the web page.

13. Click the name of the VPC you created for cloud bursting.

14. Click the name of the VPC's subnet.

15. Copy the Subnet ID.

16. Navigate to the Control browser window.

17. For Subnet ID, paste the subnet's ID copied from HUAWEI Cloud console.

18. In the HUAWEI Cloud console browser window click Security Group from the menu located onthe left-hand side of the web page.

19. Click the name of the security group you created for the VPC.

20. Copy the ID of the security group.

21. Navigate to the Control browser window.

22. For Security Group, paste the security group's ID you copied from the HUAWEI Cloud console.

23. Enable the Add public IP to VMs if you need to connect to the cloud bursting nodes from theinternet.

It is recommended to leave this disabled.

24. Navigate to the HUAWEI Cloud console browser window.

25. Click Service List.

Figure 67: HUAWEI Cloud Console

26. Under Computing, click Image Management Service.

27. Click the Private Images tab.

28. Click the name of the VM image you created for cloud bursting.

29. Copy the ID of the image.

30. Navigate to the Control browser window.

31. For OS Image URI, paste the image ID you copied from the HUAWEI Cloud console.

32. For Cloud-init script, click Browse, browse for the script file and add it.

This script will run after a cloud node is burst.

33. Add public SSH keys for any users that you want to have access to the burst nodes:

a) Login to the PBS Server.

b) Copy the public SSH key for the user.

Proprietary Information of Altair Engineering

Page 205: Control Administrator's Guide - Altair PBS Works · Altair Control 2019.1 Administrator's Guide iv • Altair Embed SE™ ©1989-2019 (formerly solidThinking Embed® SE) • Altair

Control Administrator's Guide9 Cloud Bursting p.200

Public key files are usually stored in /.ssh in the user's home directory.

c) Navigate to the Control browser window.

d) Click Add to add the public key.

e) For Public SSH keys, paste the public SSH key.

f) Repeat steps a through e to add additional public SSH keys.

The public keys are copied to each burst node for secure connectivity.

34. Click Instantiate scenario.

Proprietary Information of Altair Engineering

Page 206: Control Administrator's Guide - Altair PBS Works · Altair Control 2019.1 Administrator's Guide iv • Altair Embed SE™ ©1989-2019 (formerly solidThinking Embed® SE) • Altair

Control Administrator's Guide9 Cloud Bursting p.201

9.11 Configure Open Telekom Cloud for CloudBurstingCreate the necessary components in Open Telekom Cloud (OTC) that are required for cloud bursting andthen create a cloud bursting scenario.

Before you can begin creating the components necessary for cloud bursting, you must create andactivate an OTC Cloud account.

9.11.1 Obtain the OTC Administrator CredentialsGet the credentials for the OTC administrative user account so that you can later use this information tocreate a cloud account in Control.

You must already have signed up for an OTC account.

When subscribing to OTC, an administrative user account that has permissions for all system operationsis automatically created. Use this user account to add an OTC cloud account to Control. Reset thepassword for this account when the password is unknown.

Note: Alternately, you can create a new user and give the user administrative privileges bysetting its User Group to "admin".

1. Login to the OTC Console.

2. Click Service List.

Figure 68: Service List

3. Under Management & Deployment, click Identity and Access Management.4. Click Users from the menu located on the left-hand side of the web page.

A list of users is displayed.

5. Click located next to a username to display the user's details.The user account listed as an "admin" is the account to use to create the cloud account in Control.

Proprietary Information of Altair Engineering

Page 207: Control Administrator's Guide - Altair PBS Works · Altair Control 2019.1 Administrator's Guide iv • Altair Embed SE™ ©1989-2019 (formerly solidThinking Embed® SE) • Altair

Control Administrator's Guide9 Cloud Bursting p.202

Figure 69: User Details

6. If you do not know the password for the admin user account, click Set Credentials.

Figure 70: Set Password

a) Enable Set manually.

b) For Password, enter a password for the user account.

c) For Confirm Password, enter the password a second time.

d) Click OK.

You may have to confirm the password change either by email or by a SMS text.

9.11.2 Add an OTC AccountAdd an existing OTC Cloud account to Control so that you can create a cloud bursting scenario.

1. Login to Control.

2. Click the Cloud tab.

3. Under Infrastructure, click Cloud.

4. Click .

5. Enter the following to add a cloud account:

a) For Account name, enter any name for the cloud account.

The name can be anything meaningful to your organization, e.g., orange_cloudaccount.

b) For Auth (IAM) URL, enter: https://iam.eu-de.otc.t-systems.com/v3.

c) For Domain name, enter the OTC Domain Name you used to login to the OTC console.

d) For Username, enter a username that is an OTC administrator.

e) For Password, enter the password for the OTC administrator.

6. Click Create Account.

Proprietary Information of Altair Engineering

Page 208: Control Administrator's Guide - Altair PBS Works · Altair Control 2019.1 Administrator's Guide iv • Altair Embed SE™ ©1989-2019 (formerly solidThinking Embed® SE) • Altair

Control Administrator's Guide9 Cloud Bursting p.203

9.11.3 Check OTC Account Service QuotasView your OTC account resource usage and limits.

Quotas are used to limit the number of resources available to users. It is important to ensure you arenot exceeding your quota while setting up the resources for cloud bursting. If the existing resourcequota cannot meet your service requirements, you can submit a work order to increase your quota.Once your application is approved, OTC will update your resource quota accordingly and send you anotification.

1. Login to the OTC Console.

2. Click .

Figure 71: View Quotas

Information about resources usage and availability is displayed.

Figure 72: Resource Quotas

9.11.4 Create a Virtual Private CloudCreate a Virtual Private Cloud (VPC) so that you can perform cloud bursting.

OTC documentation for creating a VPC can be found at: Creating a VPC and Regions.

1. Login to the OTC Console.

Proprietary Information of Altair Engineering

Page 209: Control Administrator's Guide - Altair PBS Works · Altair Control 2019.1 Administrator's Guide iv • Altair Embed SE™ ©1989-2019 (formerly solidThinking Embed® SE) • Altair

Control Administrator's Guide9 Cloud Bursting p.204

2. Click Service List.

Figure 73: OTC Service List

3. Under Network, click Virtual Private Cloud.

4. Click + Create VPC.

5. In the Basic Information section:

a) For Region, select a region.

Figure 74: Regions

A region is a geographical areas and can comprise one or more availability zones (AZs).A region is completely isolated from other regions. Only AZs in the same region cancommunicate with one another through an internal network.

b) For Name, enter a name for the VPC.

c) For CIDR Block, enter an address range for the network using CIDR notation.

6. In the Subnet Settings section:

a) For Subnet Name, enter a name for the subnet.

b) For CIDR, enter an address range for the subnet using CIDR notation.

c) For Advanced Settings, click Default.7. Click Create Now.

8. Click Back to VPC List.9. Click Security Group in the left hand side menu.

10. Click + Create Security Group.

11. For Name, enter a name for the security group.

12. For Description, enter a suitable description.

13. Click OK.The security group rules are displayed.

14. Click the Inbound tab.

15. Click Add Rule.

a) For Protocol/Application, select TCP.

b) For Port, enter 22.

c) For Source, select IP Address. and enter 0.0.0.0/0.

d) Click OK.

Proprietary Information of Altair Engineering

Page 210: Control Administrator's Guide - Altair PBS Works · Altair Control 2019.1 Administrator's Guide iv • Altair Embed SE™ ©1989-2019 (formerly solidThinking Embed® SE) • Altair

Control Administrator's Guide9 Cloud Bursting p.205

9.11.5 Creating a Virtual MachineCreate a Elastic Cloud Server (ECS) using a public image provided by OTC for cloud bursting.

You must have already created a Virtual Private Network with a subnet and assigned a Security Group.If you do not have this information, see Create a Virtual Private Cloud.

OTC documentation for creating an ECS (virtual machine) can be found at Create an ECS.

1. Login to the OTC Console.

2. Click Service List.

Figure 75: OTC Service List

3. Under Computing, click Elastic Cloud Server.4. Click Key Pair from the menu located on the left-hand side of the web page.

5. Click + Create Key Pair.6. For Name, enter a name for the key pair.

7. Click OK.

8. Save the key pair (.pem) file to your local disk in a secure location.

The information in this .pem file is used later to SSH into the VM.

9. Click OK to confirm that you have downloaded the key pair file.

10. Click Elastic Cloud Server from the menu located on the left-hand side of the web page.

11. Click Create ECS.

12. For Region, select the same region that was chosen for the previously created VPC.

Figure 76: Regions

13. For AZ (Availability Zone), select an availability zone.

14. In the Specifications section:

a) Choose an ECS type category:

• General-purpose

• Dedicated general-purpose

• Memory-optimized

• Large-memory

• High-performance

• Disk-intensive

• GPU-accelerated

b) For ECS type, choose one of the flavors based on the needs of your site.

Proprietary Information of Altair Engineering

Page 211: Control Administrator's Guide - Altair PBS Works · Altair Control 2019.1 Administrator's Guide iv • Altair Embed SE™ ©1989-2019 (formerly solidThinking Embed® SE) • Altair

Control Administrator's Guide9 Cloud Bursting p.206

15. In the Image section:

a) Click Public image.

b) Select CentOS as the operating system.

c) Select Standard_CentOS7_latest(4GB) for the OS version..

16. In the Disk section, select your system disk requirements.

17. In the VPC section:

a) For VPC, select the VPC you created for cloud bursting.The NIC information is automatically populated.

b) For Security Group, select the security group you created for cloud bursting.

c) For EIP, click Automatically assignd) For Bandwidth, specify it as 5 Mbit/s.

18. For Login Mode, select Key Pair.19. For Key Pair, select the key pair file you generated earlier.

20. For Advanced Settings, click Do not configure.

21. For ECS Name, enter a name for the virtual machine.

22. For Quantity, specify 1.

23. Click Create Now.

24. Review the specifications.

25. Click Submit.26. Click Back to ECS List.

It may take some time to create the virtual machine. Once the ECS is created it is displayed in theECS list.

9.11.6 Installing and Configuring the PBS MoM on the VMInstall the PBS MoM on the virtual machine and configure the VM per your site's environment.

Log into the VM using the default user "root" and the .pem generated during the creation of the VM.

A cloud-init script is provided during the creation of the bursting scenario. This script is ran after acloud node is burst and is used to set up the system environment so that nodes burst in the cloud cancommunicate with the PBS Professional Server, therefore installation of cloud-init on the VM is required.For more information about installing cloud-init, refer to the Cloudinit Documentation and InstallingCloud-Init.

1. Login to the OTC Console.

2. Click Service List.

Figure 77: OTC Service List

3. Under Computing, click Elastic Cloud Server.

Proprietary Information of Altair Engineering

Page 212: Control Administrator's Guide - Altair PBS Works · Altair Control 2019.1 Administrator's Guide iv • Altair Embed SE™ ©1989-2019 (formerly solidThinking Embed® SE) • Altair

Control Administrator's Guide9 Cloud Bursting p.207

4. Copy the Public IP address (External IP) of the ECS.

Figure 78: VM IP Address

5. Login to your site's PBS Server.

6. SSH into the VM using the default user "root", the .pem file you generated when creating the VMand the External IP assigned to the VM.ssh -i /path/my-key-pair.pem root@IPV4PublicIP

where /path/my-key-pair.pem is the path to the .pem file downloaded while creating the virtualmachine and IPV4PublicIP is the public IP address of the virtual machine.

7. Using the PBS Professional Installation and Upgrade Guide, install and configure the PBS MOM.

8. Once the PBS MOM is installed, edit the /etc/pbs.conf file.

9. Change the value of PBS_SERVER to the hostname of the PBS Server.

10. Save the file.

11. Edit the file PBS_HOME/mom_priv/config.

where PBS_HOME is the home directory of PBS.

12. Change the value of $clienthost to the hostname of the PBS Server.

13. Save the file.

14. Configure the VM for your site's environment such as pointing it the DNS server, mounting filesystems, connecting it to the authentication service, installing any applications, etc.

15. Install cloud-init.

16. Connect the VM to your PBS server and submit a job to ensure it is working as expected.

9.11.7 Create an OS ImageCreate an image of the virtual machine configured per your site's requirements.

You must have already created a ECS (virtual machine), installed the PBS MoM on the ECS andconfigured it per your site's environment.

OTC documentation can be found at Creating a Linux Private Image.

1. Login to the OTC Console.

2. Click Service List.

Figure 79: OTC Service List

3. Under Computing, click Image Management Service.

Proprietary Information of Altair Engineering

Page 213: Control Administrator's Guide - Altair PBS Works · Altair Control 2019.1 Administrator's Guide iv • Altair Embed SE™ ©1989-2019 (formerly solidThinking Embed® SE) • Altair

Control Administrator's Guide9 Cloud Bursting p.208

4. Click + Create System Disk Image.

a) For Region, select the same region that was chosen for the previously created VPC and ECS.

Figure 80: Regions

b) For Source, click Server.c) For Server Type, click ECS.

d) For ECS, select the virtual machine created for cloud bursting.

Tip: If the virtual machine is not stopped, then stop it.

e) Click OK when prompted to verify that certain operations have been performed on the ECS.

You do not need to configure or optimize the ECS.

a) For Name, enter a name for the virtual machine.

5. Click Create Now.

6. Review the specifications.

7. Click Submit.8. Click Back to Image List.

The time required for creating an image varies depending on the image file size. Generally, ittakes about 20 minutes to create an image. The image is successfully created when its imagestatus changes to Normal.

Note: Do not perform any operation on the selected ECS or its associated resourcesduring image creation.

You can now delete the virtual machine so that you are no longer charged for it.

9.11.8 Create an OTC Cloud Bursting ScenarioCreate an OTC bursting scenario so that PBS execution nodes can be burst in the cloud.

Open a browser window and login to the OTC Console. Open a second browser window and login toControl.

A cloud-init script is provided during the creation of the bursting scenario. This script is ran after acloud node is burst and is used to set up the system environment so that nodes burst in the cloud cancommunicate with the PBS Professional Server. For more information about writing cloud-init scripts,refer to the Cloudinit Documentation and the Cloud Bursting chapter of the ControlAdministrator'sGuide.

1. Navigate to the Control browser window.

2. Click Cloud.

Proprietary Information of Altair Engineering

Page 214: Control Administrator's Guide - Altair PBS Works · Altair Control 2019.1 Administrator's Guide iv • Altair Embed SE™ ©1989-2019 (formerly solidThinking Embed® SE) • Altair

Control Administrator's Guide9 Cloud Bursting p.209

3. Under Infrastructure, click Bursting.

4. Click Add Bursting Scenario.

5. Enter information about the bursting scenario such as name, cloud provider, cloud account, andgeographic location.

a) For Name, enter a name for the cloud bursting scenario.

b) For Description, enter a description.

c) For Cloud Account, choose a previously created OTC cloud account from the list.

d) For Select a region, select the same region that was chosen for the previously created VMand VPC.

6. Click Next.Enter information about the OTC resources necessary to burst in the cloud.

7. For Domain name, provide the domain name of your organization for the cloud bursting network.altair.com, burst.altair.com

8. For Hostname prefix, enter a prefix for the names of the burst nodes.If "node" is entered , the cloud bursting nodes will be available to PBS Professional as node1,node2, node3, etc.

9. Navigate to the OTC console browser window.

10. Click Service List.

Figure 81: OTC Service List

11. Under Network, click Virtual Private Cloud.

12. Click Virtual Private Cloud from the menu located on the left-hand side of the web page.

13. Click the name of the VPC you created for cloud bursting.

14. Click the name of the VPC's subnet.

15. Copy the Subnet ID.

16. Navigate to the Control browser window.

17. For Subnet ID, paste the subnet's ID copied from the OTC console.

18. In the OTC console browser window, click Security Group from the menu located on the left-hand side of the web page.

19. Click the name of the security group you created for the VPC.

20. Copy the ID of the security group.

21. Navigate to the Control browser window.

22. For Security Group, paste the security group's ID you copied from the OTC console.

23. Enable the Add public IP to VMs if you need to connect to the cloud bursting nodes from theinternet.

It is recommended to leave this disabled.

24. Navigate to the OTC console browser window.

25. Click Service List.

Proprietary Information of Altair Engineering

Page 215: Control Administrator's Guide - Altair PBS Works · Altair Control 2019.1 Administrator's Guide iv • Altair Embed SE™ ©1989-2019 (formerly solidThinking Embed® SE) • Altair

Control Administrator's Guide9 Cloud Bursting p.210

Figure 82: OTC Service List

26. Under Computing, click Image Management Service.

27. Click the Private Images tab.

28. Click the name of the VM image you created for cloud bursting.

29. Copy the Image ID of the image.

30. Navigate to the Control browser window.

31. For OS Image URI, paste the image ID you copied from the OTC console.

32. For Cloud-init script, click Browse, browse for the script file and add it.

This script will run after a cloud node is burst.

33. Add public SSH keys for any users that you want to have access to the burst nodes:

a) Login to the PBS Server.

b) Copy the public SSH key for the user.

Public key files are usually stored in /.ssh in the user's home directory.

c) Navigate to the Control browser window.

d) Click Add to add the public key.

e) For Public SSH keys, paste the public SSH key.

f) Repeat steps a through e to add additional public SSH keys.

The public keys are copied to each burst node for secure connectivity.

34. Click Instantiate scenario.

Proprietary Information of Altair Engineering

Page 216: Control Administrator's Guide - Altair PBS Works · Altair Control 2019.1 Administrator's Guide iv • Altair Embed SE™ ©1989-2019 (formerly solidThinking Embed® SE) • Altair

Control Administrator's Guide9 Cloud Bursting p.211

9.12 Add Quotas and AlertsAdd optional resource quotas and alerts to control cloud bursting.

To control cloud bursting, quotas can be added to define bursting limits for the following resources:

• the number of CPUs

• the amount of RAM

• the amount of nodes

• the amount of node hours

Once the limit is reached, additional nodes are not burst. Alerts can also be defined so that an email issent by a SMTP server when a limit is reached.

Note: The SMTP server is configured during installation of Control by your site's systemadministrator.

1. Login to Control.

2. Click Cloud.

3. Under Infrastructure, click Bursting.

4. Click the name of the bursting scenario.

5. Click Add Quota.

6. For Resource type, choose a resource from the menu.

7. Click Add Quota.

8. For Limit, provide a limit for the resource.

When the limit is reached for the resource, the deployment of further cloud burst nodes isblocked.

9. To add an alert, click Add.

10. Provide an alert value.

When the alert value is reached, an email is sent if an SMTP server is configured.

11. Repeat steps 5 through 10 to add quotas and alerts for other resources types.

Proprietary Information of Altair Engineering

Page 217: Control Administrator's Guide - Altair PBS Works · Altair Control 2019.1 Administrator's Guide iv • Altair Embed SE™ ©1989-2019 (formerly solidThinking Embed® SE) • Altair

Control Administrator's Guide9 Cloud Bursting p.212

9.13 Add a TokenCreate a token and associated API key for a bursting scenario. The API key is used in the PBSProfessional Cloud Bursting hook.

You can create a token right after creating a bursting scenario or you can create a token at a later date.

1. Login to Control.

2. Click Cloud.

3. Under Infrastructure, click Bursting.

4. Click the name of the bursting scenario.

5. Click Add token located at the bottom of the web page.

6. For Name, enter a name for the token.

7. For Expiration date, specify the expiration date in MM/DD/YYYY format.

8. Click Add Token.An API key is generated and displayed.

9. Copy and save this API key.

The API key is only displayed once. It is pasted in the configuration file used to configure the PBSProfessional cloud bursting hook.

10. Click Close.

Once the token is created, there is additional configuration that needs to occur for PBS Professional.For more information about cloud bursting with PBS Professional 14.2.6, see the chapter Cloud Burstingwith PBS in the PBS Professional 14.2.6 Release Notes. For PBS Professional 18.2, see the chapterCloud Bursting with PBS in the PBS Professional 18.2 Administrator's Guide. For cloud bursting with PBSProfessional 18.2.2, see the Cloud Bursting chapter in the Control Administrator's Guide.

Proprietary Information of Altair Engineering

Page 218: Control Administrator's Guide - Altair PBS Works · Altair Control 2019.1 Administrator's Guide iv • Altair Embed SE™ ©1989-2019 (formerly solidThinking Embed® SE) • Altair

Control Administrator's Guide9 Cloud Bursting p.213

9.14 Configure PBS Professional for Cloud BurstingConfigure the cloud bursting hook, add custom resources for bursting, configure cloud queues and setserver cloud limits.

9.14.1 Cloud Provider Instance TypesInformation about cloud provider instance types.

The instance type (also called shapes, machine types or flavors) determines the hardware of the hostcomputer used for your cloud bursting nodes. Each instance type offers different compute, memory, andstorage capabilities. You will need to know what type of instance type you want to burst in the cloudbefore you can configure the PBS Professional. For information about instance types per cloud providersee the following table:

Table 13: Cloud Provider Instance Types

Provider URL Example Instance Types

Azure Sizes for virtual machines in Azureand High performance compute VMsizes

Standard_DS1_v2, Standard_D2s_v3, Standard_NC6

Standard_H16r, Standard_H16mr (InfiniBand)

AWS Amazon EC2 Instance Types t2.medium, r4.large, p3.2xlarge

GCP Machine Types n1-standard-8, n1-highmem-2, n1-highcpu-64

Oracle VM Shapes and GPU Instances VM.Standard1.1, VM.DenseIO1.16, VM.GPU3.1

OrangeCloud

Instance Family s1.medium, s3.large.4, cc3.large.4

HUAWEICloud

ECS Types s2.small.1, s2.medium.4, s2.xlarge.2

OTC ECS Types s1.medium, c1.large, m1.xlarge

Proprietary Information of Altair Engineering

Page 219: Control Administrator's Guide - Altair PBS Works · Altair Control 2019.1 Administrator's Guide iv • Altair Embed SE™ ©1989-2019 (formerly solidThinking Embed® SE) • Altair

Control Administrator's Guide9 Cloud Bursting p.214

9.14.2 Configure Your Network

VPNA VPN is required from the on-premise network to the cloud network for seamless communicationbetween the two.

Test the VPN setup: manually bring up the instance on the cloud and check connectivity between thelocal and cloud instances.

9.14.3 Configure Cloud Bursting Custom Resources

PBS Professional Custom Resources for Cloud BurstingCustom resources that must be added to PBS Professional to implement cloud bursting.

cloud_instance_typeQueue-level stringCloud provider instance type (machine, shape type or flavor) associated to the queue.

cloud_min_instancesServer-level integer.Minimum number of instances (nodes) to be present in the cloud at any time.

cloud_max_instancesServer-level integer.Maximum number of instances (nodes) that can be made available in the cloud.

cloud_max_jobs_check_per_queueQueue-level integer.The number of queued jobs in the cloud queue that are checked to determine the number ofnodes that must be burst based on the instance type requested.Default is 64. Must be greater than zero. Setting it to zero results in no jobs being considered forcloud bursting in that queue.

cloud_networkHost-level stringUsed to group a set of nodes with the same network name. Used to request InfiniBand enablednodes.

cloud_node_imageHost-level stringOS image that is to be used when a cloud node is burst.

cloud_node_instance_typeHost-level stringIndicates the default instance type of the cloud node. Is set by the cloud bursting hook when thenode is burst.Node-level string

Proprietary Information of Altair Engineering

Page 220: Control Administrator's Guide - Altair PBS Works · Altair Control 2019.1 Administrator's Guide iv • Altair Embed SE™ ©1989-2019 (formerly solidThinking Embed® SE) • Altair

Control Administrator's Guide9 Cloud Bursting p.215

Indicates the instance type of the cloud node requested at job submission. This overridesthe default instance type set at the queue-level. A cloud node is burst with this instance typesignature and the job is directed to that cloud node. Is set by the cloud bursting hook when thenode is burst.

cloud_provisioned_timeHost-level integer.Time at which the cloud node is created.

cloud_queueQueue-level BooleanIndicates whether the queue is a cloud queue. When True, the queue is a cloud queue.

cloud_scenarioNode-level, Server-level string.Indicates the associated scenario type for the queue and the node.

idle_before_unburstServer-level duration. Is expressed as an integer whose units are in seconds.Minimum time that a cloud node can be idle before it is unburst.

lic_signatureHost-level string.Contains node hour based information for licensing

node_locationHost-level string.Used to differentiate local nodes from cloud nodes. It is useful when the site wants the schedulerto try to place jobs on the on-premise nodes first before trying to run them on cloud nodes, whilerestricting jobs from running on both. Used with placement sets.

Further Explanation for cloud_max_jobs_check_per_queueThe following example further explains the use of cloud_max_jobs_check_per_queue.

1. The cloud hook first checks for queued jobs in the cloud queue.

2. It then checks the number of resources it requires to run those jobs.

3. It adds up all the resources required for the queued jobs to run.

4. It calculates the number of nodes required to run the job.

Example:

Suppose there is a cloud queue and the instance type specified for the queue is Standard_DS2_V2. Theconfiguration for Standard_DS1_v2 is{“ncpus”:2, “mem”:”8gb”}.

Jobs that are in a queued state in the cloud queue are J1, J2, and J3.

• J1 needs 3 CPUs to run.

• J2 needs 2 CPUs to run.

• J3 needs 5 CPUs to run.

The number of CPUs required to run these jobs is 3+2+5 = 10 CPUs.

Proprietary Information of Altair Engineering

Page 221: Control Administrator's Guide - Altair PBS Works · Altair Control 2019.1 Administrator's Guide iv • Altair Embed SE™ ©1989-2019 (formerly solidThinking Embed® SE) • Altair

Control Administrator's Guide9 Cloud Bursting p.216

The number of nodes needed to run all 3 jobs = (10 CPUs/2[Standard_DS2_V2 number of CPUs]) = 5nodes are needed to burst nodes of type Standard_DS2_V2.

cloud_max_jobs_check_per_queue is set to the default value of 64. If 100 jobs are queued in the cloudqueue (J1.….J100), then only the first 64 jobs (J1.…..J64) are used to calculate how many nodes needto be burst. You may change the value of cloud_max_jobs_check_per_queue to tune cloud bursting.

Create Cloud Bursting Custom ResourcesAdd custom resources required to implement cloud bursting.

Creating custom resources must be done as root or as a user with sudo permissions using the sudocommand.

1. Login to the PBS Server.

2. Copy and execute the following commands to create the custom resources required for cloudbursting:qmgr -c "create resource cloud_queue type=boolean" qmgr -c "create resource cloud_instance_type type=string"qmgr -c "create resource cloud_node_instance_type type=string,flag=h"qmgr -c "create resource cloud_min_instances type=long"qmgr -c "create resource cloud_max_instances type=long"qmgr -c "create resource cloud_provisioned_time type=long,flag=h"qmgr -c "create resource lic_signature type=string,flag=h"qmgr -c "create resource cloud_node_image type=string,flag=h"qmgr -c "create resource cloud_network type=string,flag=h"qmgr -c "create resource node_location type=string,flag=h"qmgr -c "create resource cloud_max_jobs_check_per_queue type=long"qmgr -c "create resource cloud_scenario type=string,flag=h"qmgr -c "create resource idle_before_unburst type=long"

3. Navigate to PBS_HOME/sched_priv.

4. Edit the sched_config file.

5. Add cloud_scenario, cloud_node_image, cloud_node_instance_type, and cloud_network tothe resources line:resources: "ncpus, mem, arch, host, vnode, netwins, aoe, cloud_scenario, cloud_node_image, cloud_node_instance_type, cloud_network"

Proprietary Information of Altair Engineering

Page 222: Control Administrator's Guide - Altair PBS Works · Altair Control 2019.1 Administrator's Guide iv • Altair Embed SE™ ©1989-2019 (formerly solidThinking Embed® SE) • Altair

Control Administrator's Guide9 Cloud Bursting p.217

9.14.4 Set Cloud Limits at ServerSet server limits for the minimum and maximum number of cloud nodes and minimum uptime andminimum idle time for a cloud node.

Setting server limits must be done as root or as a user with sudo permissions using the sudo command.

1. Login to the PBS Server.

2. Copy and execute the following commands to set server limits:qmgr -c "set server resources_available.cloud_min_instances = 0"qmgr -c "set server resources_available.cloud_max_instances = MAX_INSTANCES"qmgr -c "set server resources_available.idle_before_unburst = MIN_IDLE_TIME

Where MAX_INSTANCES is the maximum number of nodes that can be made available in thecloud and must be expressed as an integer greater than zero. MIN_IDLE_TIME is the minimumtime that a cloud node can be idle before it is unburst and must be expressed as an integer whoseunits are in seconds.

Note: Recommendation is to set the minimum idle time to at least 15 minutes, testcloud bursting and adjust as necessary.

9.14.5 Configure the Cloud QueuesAdd and configure the queues necessary for cloud bursting.

Creating and configuring a queue must be done as root or as a user with sudo permissions using thesudo command.

The type of node that is burst is defined at the queue-level, unless overridden at job submission time.

To support poly-cloud bursting, the cloud bursting hook has been reformatted allowing scenarios to beadded for each cloud provider. A cloud queue must be created for each scenario that is defined in thecloud bursting hook configuration file. Additionally, multiple cloud queues can be created for the samebursting scenario having different instance types allowing different machine types to be burst per cloudprovider.

1. Login to the PBS Server.

2. Copy and execute the following commands to create and configure the cloud queue:qmgr -c "create queue QUEUE_NAME queue_type=execution,enabled=true,started=true"qmgr -c "set queue QUEUE_NAME resources_available.cloud_queue = True"

Where QUEUE_NAME is the name of the queue.

3. Optionally, set the maximum number of jobs to check in the queue to determine the number ofnodes to burst:qmgr -c "set queue QUEUE_NAME \resources_available.cloud_max_jobs_check_per_queue = MAX_JOBS"

Where MAX_JOBS is the number of queued jobs in the cloud queue that are checked to determinethe number of nodes that must be burst based on the instance type requested.

Proprietary Information of Altair Engineering

Page 223: Control Administrator's Guide - Altair PBS Works · Altair Control 2019.1 Administrator's Guide iv • Altair Embed SE™ ©1989-2019 (formerly solidThinking Embed® SE) • Altair

Control Administrator's Guide9 Cloud Bursting p.218

Default is 64. MAX_JOBS must be greater than zero. Setting it to zero results in no jobs beingconsidered for cloud bursting in that queue.

4. Assign a bursting scenario to the queue.qmgr -c "set queue QUEUE_NAME resources_available.cloud_scenario = SCENARIO_NAME"

Where SCENARIO_NAME is the name of the scenario as defined by the cloud bursting hookconfiguration file. Cloud nodes are burst based on this scenario.

Hook configuration file scenario:"scenario": { "azure_scenario_1": { "api_key": "API-KEY-HERE", "cloud_default_image": "azure_bursting_image1", "instance_types":{ "Standard_DS2_v2": {"ncpus": 2, "mem": "7gb" }, "Standard_DS3_v2": {"ncpus": 4, "mem": "14gb" }, "Standard_DS1_v2": {"ncpus": 1, "mem": "3.5gb"} }, "cloud_max_instances": 10, "max_vms_for_infiniband_scaleset" : 100, "max_nodes_per_burst":50, "cloud_node_wait_timeout":180 } }

Associate the scenario with the queue using the command:qmgr -c "set queue QUEUE_NAME \resources_available.cloud_scenario = azure_scenario_1"

5. Execute the following command to associate an instance type to the queue:qmgr -c "set queue QUEUE_NAME \resources_available.cloud_instance_type = INSTANCE_TYPE"

Where INSTANCE_TYPE is the name of cloud provider instance type. This is the type of machinethat will be burst. It must match one of the instance types defined for the scenario that was justassigned to the queue.qmgr -c "set queue QUEUE_NAME \resources_available.cloud_instance_type = Standard_DS2_v2"

9.14.6 Install the PCLM CLI and the Cloud Bursting Hook

PCLM CLI and the Cloud Bursting HookComponents that must be installed on the PBS Server to enable cloud bursting.

Two components that must be installed on the PBS Server to enable cloud bursting are the PCLM CLIand a cloud bursting hook.

PCLM CLIThe PCLM CLI is a command line interface for bursting and unbursting cloud nodes that is used inconjunction with a cloud bursting hook to enable cloud bursting.

Proprietary Information of Altair Engineering

Page 224: Control Administrator's Guide - Altair PBS Works · Altair Control 2019.1 Administrator's Guide iv • Altair Embed SE™ ©1989-2019 (formerly solidThinking Embed® SE) • Altair

Control Administrator's Guide9 Cloud Bursting p.219

The PCLM CLI can also be used to test bursting and unbursting of cloud nodes without using the cloudbursting hook. You can burst one VM, several VMs of the same type, or several VMs of different types inone single bursting call.

The current version of the PCLM CLI client is 0.4.6. It is compatible with Control 2019.1 and PBSProfessional 19.2.1.

Cloud Bursting HookA periodic hook that monitors cloud queues, estimates the demand for cloud nodes, and dynamicallyadjusts the number of nodes by bursting or unbursting as needed.

Location of the PCLM CLI and the Cloud Bursting HookThe PCLM CLI and the cloud bursting hook are contained in a tarball file that is available after installingControl 2019.1. The tarball file is located at:

PC_HOME/config/shared/pbspro_cloud_bursting_hook.tar.gz

Structure of the Cloud Bursting Tarball FileBelow is the expected directory structure after extracting the contents ofpbspro_cloud_bursting_hook.tar.gz. The install.sh script is run to install both the PCLM CLI andthe cloud bursting hook.

total 68drwxr-xr-x 4 495 491 86 Mar 18 21:39 pbspro-cloud-bursting-rw-r--r-- 1 495 491 67333 Mar 18 21:39 pclm_cli-0.4.6.tgz

./pbspro-cloud-bursting:total 16drwxr-xr-x 2 495 491 60 Mar 18 21:39 cloud_hook-rwxr-xr-x 1 495 491 11242 Mar 18 21:39 install.sh-rw-r--r-- 1 495 491 100 Mar 18 21:39 README.mddrwxr-xr-x 2 495 491 18 Mar 18 21:39 scripts

./pbspro-cloud-bursting/cloud_hook:total 104-rw-r--r-- 1 495 491 1618 Mar 18 21:39 cloud_burst_hook.CF-rw-r--r-- 1 495 491 102102 Mar 18 21:39 cloud_burst_hook.PY

./pbspro-cloud-bursting/scripts:total 4-rw-r--r-- 1 495 491 1252 Mar 18 21:39 pclm

Install PCLM CLI and Cloud Bursting HookCopy the tarball containing the PCLM CLI and cloud bursting hook, extract the contents and install thePCLM CLI and cloud bursting hook.

Before installing the PCLM CLI and the cloud bursting hook, uninstall any previous installations.

Installation must be done as root or as a user with sudo permissions using the sudo command.

1. Copy the below file from the machine where Control 2019.1 is installed to the PBS Server using acommand such as scp:

Proprietary Information of Altair Engineering

Page 225: Control Administrator's Guide - Altair PBS Works · Altair Control 2019.1 Administrator's Guide iv • Altair Embed SE™ ©1989-2019 (formerly solidThinking Embed® SE) • Altair

Control Administrator's Guide9 Cloud Bursting p.220

PC_HOME/config/shared/pbspro_cloud_bursting_hook.tar.gz

2. Login to the PBS Server.

3. Navigate to where the pbspro_cloud_bursting_hook.tar.gz tarball file was copied.

4. Extract the contents of the tarball file.tar xvf pbspro_cloud_bursting_hook.tar.gz

5. Navigate to the directory containing install.sh.cd pbspro-cloud-bursting

6. Install the PCLM CLI and the cloud bursting hook:./install.sh --type install --pclm-location ../pclm_cli-0.4.6.tgz

7. Test the installation of the PCLM CLI.pclm --help

See AlsoPCLM CLI and the Cloud Bursting Hook

9.14.7 Test Cloud Bursting with the CLITest cloud bursting using the PCLM CLI before trying to burst nodes using the cloud bursting hook.This will ensure that the connectivity from the PBS Server to the cloud infrastructure and the burstingscenario are working properly.

Before you begin testing cloud bursting:

1. Ensure that the cloud infrastructure (VPN connectivity, networks, firewalls etc.) is configured.

2. The cloud account and bursting scenario must be configured in Control.

3. A bursting token (API Key) for the scenario must be generated.

4. The PCLM CLI must be installed.

Burst an Execution Node in the CloudUse the PCLM CLI client to burst cloud nodes.

For more information about cloud provider instance/machine types see Cloud Provider Instance Types.

Cloud bursting can be accomplished from the command line by issuing a single command. You can thenget the status of the bursting activity and also capture the output in raw JSON format.

1. Execute the below command to burst a node in the cloud:pclm --api-endpoint=http://HOSTNAME:9980/control/ --api-key API_KEY \bootstrapper --wait scenario \burst '{"mom":[{"deployable_id": "MACHINE_TYPE", "count":COUNT}]}'

Where:

• HOSTNAME is the IP address or hostname of the machine where Cloud is installed.

• API_KEY is the API Key generated for a cloud bursting scenario.

• boostrapper is the microservices module for bursting.

Proprietary Information of Altair Engineering

Page 226: Control Administrator's Guide - Altair PBS Works · Altair Control 2019.1 Administrator's Guide iv • Altair Embed SE™ ©1989-2019 (formerly solidThinking Embed® SE) • Altair

Control Administrator's Guide9 Cloud Bursting p.221

• --wait causes the application to wait for the node to be burst.

• MACHINE_TYPE is the name of cloud provider instance type. This is the type of machine thatwill be burst. It must match one of the instance types defined by the cloud provider.

• COUNT indicates the number of nodes to burst.

2. Execute this command to view the status of the cloud bursting activity.pclm --api-endpoint=http://HOSTNAME:9980/control/ --api-key API_KEY \bootstrapper scenario status

The status is displayed as a table. Use --raw to display the output of the command in JSONformat.

The status command will display the machine id, machine private ip and hostname.

The states are:

• RUNNING indicates deployment ongoing.

• SUCCESS indicates deployment finished, machine is ready

• CREATED indicates machines are created. This status is short-lived and rarely visible.

• ERROR indicates that something has gone wrong and machine deployment has failed.

• UNDEPLOYMENT RUNNING indicates that machines are being removed.

• UNDEPLOYMENT ERROR indicates that something has gone wrong while machine was beingremoved.

When an undeployment of a machine is finished, it disappears from the status table.

Unburst an Execution Node in the CloudUse the PCLM CLI client to unburst cloud nodes.

Use the PCLM CLI to view the status of the nodes while cloud bursting is in progress. Execute theunbursting command to remove cloud nodes.

1. Execute this command to get the status of the cloud bursting activity.pclm --api-endpoint=http://HOSTNAME:9980/control/ --api-key API_KEY \bootstrapper scenario status

Where:

• HOSTNAME is the IP address or hostname of the machine where Cloud is installed.

• API_KEY is the API Key generated for the cloud bursting scenario.

The status command will display the machine id, machine private ip and hostname for the nodes.

2. Execute this command for unbursting.pclm --api-endpoint=http://HOSTNAME:9980/control/ --api-key API_KEY \bootstrapper --wait scenario \unburst MACHINE_ID|MACHINE_PRIVATE_IP|HOSTNAME

Where:

• MACHINE_ID is the ID of the machine to be removed.

• MACHINE_PRIVATE_IP is the IP of the machine to be removed.

Proprietary Information of Altair Engineering

Page 227: Control Administrator's Guide - Altair PBS Works · Altair Control 2019.1 Administrator's Guide iv • Altair Embed SE™ ©1989-2019 (formerly solidThinking Embed® SE) • Altair

Control Administrator's Guide9 Cloud Bursting p.222

• HOSTNAME is the hostname of the machine to be removed.

You can use ID, Private IP or hostname of a machine to unburst it.

9.14.8 Configure the Cloud Bursting HookConfigure the PBS Professional cloud bursting hook that monitors cloud queues, estimates the demandfor cloud nodes, and dynamically adjusts the number of nodes by bursting or unbursting as needed.

Cloud Bursting Periodic HookA periodic hook that monitors cloud queues, estimates the demand for cloud nodes, and dynamicallyadjusts the number of nodes by bursting or unbursting as needed.

The cloud bursting hook has been changed so that cloud nodes may be burst across multiple cloudproviders. This is useful when your site wants to for instance burst InfiniBand nodes which are currentlysupported by Azure, but you also want to burst spot instance nodes in AWS. To support this newfeature, the cloud bursting hook configuration file now includes a section called scenario. A scenariomust be added to the cloud bursting hook configuration file for each cloud provider.

Addtionally, the DNS setup that was required for previous versions of the cloud bursting hook is nolonger required. Cloud nodes are accessed through their IP address, therefore the DNS information inthe hook configuration file has been removed.

The cloud bursting configuration file contains the following parameters:

pclm_serverEndpoint for accessing Cloud in the format http://hostname:port or http://ipaddress:port.Default port is 9980.

idle_before_unburstMinimum time that a cloud node can be idle before it is unburst. Is expressed as an integer whoseunits are in seconds.This value is overridden by the same value set at the server-level.

cloud_min_instances

Minimum number of instances to be present in the cloud at any time. Does not apply duringstartup; cloud nodes are not immediately burst on startup. This is the minimum number that aremaintained after they are initially burst on demand. This value is overridden by the same valueset at the server-level.

resourcesResources to be considered for calculating the number of nodes to burst. Resources that areprovided for this parameter should also be added to the signature of each instance_type asdefined by the instance_types parameter. Resources added should be of numeric type.Currently, resources that are supported are: "ncpus", "mem", "ngpus".

Proprietary Information of Altair Engineering

Page 228: Control Administrator's Guide - Altair PBS Works · Altair Control 2019.1 Administrator's Guide iv • Altair Embed SE™ ©1989-2019 (formerly solidThinking Embed® SE) • Altair

Control Administrator's Guide9 Cloud Bursting p.223

scenarioContainer for bursting scenarios. You can have scenarios for each cloud provider or multiplescenarios for a cloud provider or both.

scenario_1...n

api_keyScenario token generated for the bursting scenario.

cloud_default_imageOS image that is to be used by default when a cloud node is burst. This is overriddenwhen the OS image is provided at job submission via the qsub command.

instance_typesContainer for providing information about cloud provider instance types (machinetypes,shapes or flavors). Instance types are defined by the cloud provider. The nameof the instance type and the CPU, GPU and memory signature must exactly match thecloud provider's.

cloud_max_instances

Maximum number of instances that can be made available in the cloud for thisscenario. Is expressed by an integer and must be greater than 0. This value isoverridden by the resource value set at the server-level.

max_vms_for_infiniband_scaleset (Optional)Defines the number of nodes on an single InfiniBand switch. Supported by Azure only.This limit is set by Azure and the current default is 100. Customers may negotiatewith Azure to increase this limit. In this case, the value of this parameter should beupdated.

max_nodes_per_burst

Maximum number of nodes allowed to burst in a single hook cycle.

Maximum number of cloud node licenses to renew per hook cycle.

cloud_node_wait_timeoutMaximum time to wait for freshly burst node to become usable. Is expressed as aninteger whose units are in seconds. Default is 180 seconds. You can set this to ahigher value, but a lower value defaults to 180 seconds.

preemptable (Optional)This flag is set to True if the scenario is supporting preemptible (spot) instances.Default value is False for a scenario. If flag is not specified, the default value is used.Useful when using cloud spot instances for running short re-runnable jobs that couldbe preempted.

Proprietary Information of Altair Engineering

Page 229: Control Administrator's Guide - Altair PBS Works · Altair Control 2019.1 Administrator's Guide iv • Altair Embed SE™ ©1989-2019 (formerly solidThinking Embed® SE) • Altair

Control Administrator's Guide9 Cloud Bursting p.224

Default Cloud Bursting Hook Configuration File{ "pclm_server": "pclm_server:port_number", "idle_before_unburst": 180, "cloud_min_instances": 1, "resources":["ncpus", "mem", "ngpus"], "scenario": { "azure_scenario_1": { "api_key": "API-KEY-HERE", "cloud_default_image": "<CloudDefaultImage>", "instance_types":{ "Standard_DS2_v2": {"ncpus": 2, "mem": "7gb" }, "Standard_DS3_v2": {"ncpus": 4, "mem": "14gb" }, "Standard_DS1_v2": {"ncpus": 1, "mem": "3.5gb"} }, "cloud_max_instances": 10, "max_vms_for_infiniband_scaleset" : 100, "max_nodes_per_burst":50, "cloud_node_wait_timeout":180 }, "aws_scenario_1": { "api_key": "API-KEY-HERE", "cloud_default_image": "<CloudDefaultImage>", "instance_types":{ "t2.small": {"ncpus": 1, "mem": "2gb" }, "t2.micro": {"ncpus": 1, "mem": "1gb"} }, "cloud_max_instances": 20, "max_nodes_per_burst":50, "cloud_node_wait_timeout":180 }, "aws_scenario_2": { "api_key": "API KEY HERE", "cloud_default_image": "<CloudDefaultImage>", "instance_types":{ "t2.large": {"ncpus": 2, "mem": "8gb" }, "t2.xlarge": {"ncpus": 4, "mem": "16gb" }, "t2.2xlarge": {"ncpus": 8, "mem": "32gb"} }, "cloud_max_instances": 50, "max_nodes_per_burst":50, "cloud_node_wait_timeout":180 } }}

Configure the Cloud Bursting HookConfigure the cloud bursting hook per your site's requirements and then re-import it.

Configuring the hook must be done as root or as a user with sudo permissions using the sudocommand.

Export the cloud bursting hook configuration file, update it's parameters per your site's requirements,and then re-import the hook.

1. Export the cloud bursting hook configuration to a file:

Proprietary Information of Altair Engineering

Page 230: Control Administrator's Guide - Altair PBS Works · Altair Control 2019.1 Administrator's Guide iv • Altair Embed SE™ ©1989-2019 (formerly solidThinking Embed® SE) • Altair

Control Administrator's Guide9 Cloud Bursting p.225

qmgr -c "export hook cloud_hook application/x-config default" > FILENAME

Where FILENAME is the name of the file where the hook configuration is written.

qmgr -c "export hook cloud_hook application/x-config default" \> config.json

2. Edit the file.

3. Set pclm_server to the endpoint for accessing Cloud in the format http://HOSTNAME:9980.

Where HOSTNAME is the IP address or hostname of the machine where Cloud is installed.

4. Set idle_before_unburst to the minimum time that a cloud node can be idle before it is unburst.

Must be expressed as an integer whose units are in seconds.

5. Set the value of cloud_min_instances to the minimum number of instances to be present in thecloud at any time.

This value is overridden by the same value set at the server-level.

6. Set resources to a comma separated list of resources that are to be considered for calculating thenumber of nodes to burst.["ncpus", "mem", "ngpus"] OR ["ncpus", "mem"]

7. For each scenario to be added to the scenario container:

Note: Scenarios are specific to a cloud provider.

a) Set the value of api_key to the token generated for the bursting scenario.

b) Set the value of cloud_default_image to OS image that should be used for bursting.

This OS image is used if the image is not requested as part of the job submission request.

c) For the instance_types container, add a comma separated list of any instance types yoursite may want to burst.

Instance types are defined by the cloud provider. The name of the instance type and the CPU,GPU and memory signature must exactly match the cloud provider's."instance_types":{ "t2.large": {"ncpus": 2, "mem": "8gb" }, "t2.xlarge": {"ncpus": 4, "mem": "16gb" }, "t2.2xlarge": {"ncpus": 8, "mem": "32gb"} },

d) Set the value of cloud_max_instances to the maximum number of instances that can bemade available in the cloud.

Must be expressed an integer greater than zero. This value is overridden by the resourcevalue set at the server-level.

e) Optionally, set the value of max_vms_for_infiniband_scaleset to define the maximumnumber of nodes allowed on a single InfiniBand switch.

This value should match the value of Maximum number of VMs inside a ScaleSet as specifiedin the Azure bursting scenario. If you are not using InfiniBand, then you can eliminate thisparameter.

Note: Supported by Azure only.

Proprietary Information of Altair Engineering

Page 231: Control Administrator's Guide - Altair PBS Works · Altair Control 2019.1 Administrator's Guide iv • Altair Embed SE™ ©1989-2019 (formerly solidThinking Embed® SE) • Altair

Control Administrator's Guide9 Cloud Bursting p.226

f) Set the value of max_nodes_per_burst to the maximum number of nodes allowed to burst ina single hook cycle.

g) Set the value of cloud_node_wait_timeout to the maximum time to wait for freshly burstnode to become usable.

Must be expressed as an integer whose units are in seconds. Default is 180 seconds. You canset this to a higher value, but a lower value defaults to 180 seconds.

h) Set preemptable to:

• True when the bursting scenario supports preemptible (spot) instances and cloud jobsmay be preempted.

• False when jobs that are run in the cloud should not be preemptible.

Default value is False for a scenario. If flag is not specified, the default value is used.

8. Re-import the hook with the updated configuration values:qmgr -c "import hook cloud_hook application/x-config default FILENAME"

Where FILENAME is the name of the file containing the hook configuration.qmgr -c "import hook cloud_hook application/x-config default config.json"

9. Optionally, set the cloud bursting hook frequency.

The default frequency for the cloud bursting hook to execute is every 2 minutes.

qmgr -c "set hook cloud_hook freq=FREQUENCY"

Where FREQUENCY is the frequency for the hook to execute. Must be expressed as an integerwhose units are in seconds.

10. Set the cloud bursting hook alarm:

You must set the hook alarm; otherwise, the hook will time out after the default value of 600seconds. Tune the alarm value to something greater than the default. Consider the followingfactors:

• Time required to burst nodes on the cloud

• Time required to unburst nodes on the cloud

• Number of cloud queues

Tip: Recommendation is to set the alarm to at least 20 minutes (1200 seconds).

qmgr -c "set hook cloud_hook alarm=ALARM_TIME"

Where ALARM_TIME must be expressed as an integer whose units are in seconds.

11. Enable the cloud bursting hook.

The cloud bursting server periodic hook is disabled by default.qmgr -c "set hook cloud_hook enabled=True"

Proprietary Information of Altair Engineering

Page 232: Control Administrator's Guide - Altair PBS Works · Altair Control 2019.1 Administrator's Guide iv • Altair Embed SE™ ©1989-2019 (formerly solidThinking Embed® SE) • Altair

Control Administrator's Guide9 Cloud Bursting p.227

9.14.9 Test Cloud Bursting by Submitting a JobTest cloud bursting by submitting jobs to cloud queues.

Tip: To troubleshoot issues with the cloud bursting hook, specify that all log events shouldbe captured in the PBS Server logs. Make sure to reset this value after testing is complete.qmgr -c "set server log_events = 2047"

1. Submit jobs to the cloud queue.qsub -l select=1:ncpus=4 -q CLOUD_Q JOB_SCRIPTqsub -l select=1:ncpus=4 -q CLOUD_Q JOB_SCRIPT

Where:

• CLOUD_Q is the name of the cloud queue.

• JOB_SCRIPT is the name of the script to be executed.

2. Check the status of the jobs:qstat -s

3. Tail the PBS Professional server logs.tail -f PBS_HOME/server_logs/CURRENT_SERVER_LOG_FILE

Where CURRENT_SERVER_LOG_FILE is the most current PBS Server log file.

4. Check the current log file to verify that the cloud bursting hook is started.

Search for the name of the cloud bursting hook.Server@master;Hook;HOOK_NAME;started

Where HOOK_NAME is the name of the cloud bursting hook.

5. Check the status of the nodes to verify that PBS Professional burst to cloud nodes:pbsnodes -av

6. Check the status of the jobs.

Once the node is added, jobs should start running.qstat -s

Proprietary Information of Altair Engineering

Page 233: Control Administrator's Guide - Altair PBS Works · Altair Control 2019.1 Administrator's Guide iv • Altair Embed SE™ ©1989-2019 (formerly solidThinking Embed® SE) • Altair

Control Administrator's Guide9 Cloud Bursting p.228

9.15 Troubleshoot Cloud BurstingView log messages to troubleshoot cloud bursting.

View the log messages that are written to PBS_HOME/server_logs on the PBS Server to troubleshootcloud bursting. Additionally, you can SSH to the virtual machine that was burst and check PBS_HOME/mom_logs to debug issues with node creation or issues with starting the MoM on that virtual machine.

A third option for troubleshooting cloud bursting is to use Control to view the logs through a Kibanauser interface.

PBS MoM's Stopped or DownWhen cloud bursting is implemented and all PBS MoM's are stopped or down, error messages similar tothe below may be seen in the PBS Server logs:

Server@server;Hook;Server@server;CLBR: Error: /opt/pbs/bin/pbsnodes: Server has no node listServer@server;Hook;Server@server;CLBR: Error: Failed to get nodes info

Resolve the issue by starting at least one MoM.

See AlsoAccess the Cloud Logs

Proprietary Information of Altair Engineering

Page 234: Control Administrator's Guide - Altair PBS Works · Altair Control 2019.1 Administrator's Guide iv • Altair Embed SE™ ©1989-2019 (formerly solidThinking Embed® SE) • Altair

Control Administrator's Guide9 Cloud Bursting p.229

9.16 Manage Cloud and On-Premise JobsSolutions to handle various job distribution needs.

Your site may want to run certain types of jobs on-premises or in the cloud. PBS Professional providesvarious methods to collect and distribute jobs. For more information see Routing Jobs in the PBSProfessional Administrator’s Guide. Here are some solutions to handle various job distribution needs.

Associate Nodes with QueuesIn general, start by associating nodes with queues, so that a job in a specific queue is sent to thedesired nodes. Associate each on-premise vnode with the appropriate local queue. Set the queueattribute on the vnode to the name of the local queue:

qmgr -c "active <list of on-premise nodes>"qmgr -c "set node queue=<name of local queue>"

For more information see Creating Queues and Associating Vnodes With One Queue in the PBSProfessional Administrator’s Guide.

Use Hooks or Routing Queues to Send Jobs to the Appropriate QueueTo send jobs to the appropriate queue on submission, use hooks or routing queues. For moreinformation see the PBS Professional Hooks Guide and Routing Queues in the PBS ProfessionalAdministrator’s Guide.

Use Peering to Pull Jobs to Empty NodesTo allow jobs to be run where resources are available, use peering to pull jobs to empty nodes. Formore information see Peer Scheduling in the PBS Professional Administrator’s Guide.

9.16.1 Job Distribution Examples and SolutionsExamples for routing jobs to the cloud.

Burst when On-premise Nodes are Not AvailableYou want to run as many jobs as possible on on-premise nodes, and only burst to the cloud when nomore on-premise nodes are available.

Solution: Create two execution queues, one for local jobs and one for cloud jobs. Set up local peeringso that the local queue pulls from the cloud queue whenever local nodes are available.

Send Small Jobs to the CloudYou have big machines for on-premise nodes and want to reserve those big machines for big jobs. Youwant to send smaller jobs to the cloud.

Solution 1:

• Create three queues: a routing queue, a local queue, and a cloud queue.

• Use the routing queue to collect jobs on submission.

• Set resource gating on the local queue to filter out smaller jobs.

Proprietary Information of Altair Engineering

Page 235: Control Administrator's Guide - Altair PBS Works · Altair Control 2019.1 Administrator's Guide iv • Altair Embed SE™ ©1989-2019 (formerly solidThinking Embed® SE) • Altair

Control Administrator's Guide9 Cloud Bursting p.230

• Allow smaller jobs into the cloud queue.

Solution 2:

• Create two queues: a local queue, and a cloud queue.

• Use a queuejob hook to route jobs into the appropriate queue.

Send Specific Jobs Only to the CloudYou want to send specific jobs to the cloud because:

• an application needed by some jobs runs well in the cloud.

• a resource that is available in the cloud is not available locally.

• a department has exhausted its share of local resources, and wants to send its jobs to the cloud.

Solution :

• Create two queues: a local queue, and a cloud queue.

• Use a queuejob hook to route jobs into the appropriate queue.

Charge Departments for Resources UsedYou have multiple departments and each department should be charged for the resources it uses.

Solution :

• Create a cloud queue for each department.

• Set separate limits on each cloud queue.

Proprietary Information of Altair Engineering

Page 236: Control Administrator's Guide - Altair PBS Works · Altair Control 2019.1 Administrator's Guide iv • Altair Embed SE™ ©1989-2019 (formerly solidThinking Embed® SE) • Altair

Control Administrator's Guide9 Cloud Bursting p.231

9.17 Run Cloud Jobs On-Premise Before BurstingUse placement sets to run jobs on-premise first and burst cloud nodes when local nodes are notavailable.

Assign a bursting scenario to an on-premise vnode so that cloud jobs are run on-premise when there iscapacity. Cloud nodes are only burst when on-premise nodes are not available.

For more information see Placement Sets in the PBS Professional Administrator’s Guide.

1. Create a custom resource to indicate that a node is on-premise.qmgr -c "create resource node_location type=string,flag=h"

2. For each on-premise vnode, set the resources_available.node_location resource to "local" toindicate that it is on-premise.qmgr -c "active node VNODE_NAME"qmgr -c "set node VNODE_NAME resources_available.node_location=local"

Where VNODE_NAME is the name of the local vnode.

3. For any on-premise vnodes where you want to run cloud jobs, associate a bursting scenario or alist of bursting scenarios to the vnode:

• To associate a list of bursting scenarios to a vode:qmgr -c "set node VNODE_NAME \resources_available.cloud_scenario= SCENARIO_1, SCENARIO_2,...,SCENARIO_n"

• To associate a single bursting scenario to a vode:qmgr -c "set node VNODE_NAME resources_available.cloud_scenario= SCENARIO_1"

Where SCENARIO_1,..., SCENARIO_n are scenario names that have been added to the cloudbursting hook configuration file.qmgr -c "set node VNODE_NAME \resources_available.cloud_scenario= azure_scenario_1,aws_scenario_1,aws_scenario_2"

4. Create a dummy vnode with a high resources_available.ncpus value and unsetresources_available.node_location.

5. Mark the dummy node as offline so that jobs cannot run on it.pbsnodes -C "COMMENT" -o HOSTNAME

Where COMMENT is a string explaining that this is a dummy node for cloud bursting andHOSTNAME is the hostname of the vnode.

6. Enable placement sets:qmgr -c "set server node_group_enable=true"qmgr -c "set server node_group_key=node_location"

7. Allow creation of placement sets from unset resources:qmgr -c "set sched only_explicit_psets=false"

8. Optionally, prevent any single job from running on both local and cloud vnodes:qmgr -c "set sched do_not_span_psets=true"

9. Navigate to PBS_HOME/sched_priv.

10. Edit the sched_config file.

11. Add cloud_scenario and node_location to the resources line:"... , cloud_scenario, node_location"

Proprietary Information of Altair Engineering

Page 237: Control Administrator's Guide - Altair PBS Works · Altair Control 2019.1 Administrator's Guide iv • Altair Embed SE™ ©1989-2019 (formerly solidThinking Embed® SE) • Altair

Control Administrator's Guide9 Cloud Bursting p.232

12. For each local (non-cloud) queue set its node_location to "local":qmgr -c "set queue QUEUE_NAME resources_default.node_location=local"

Where QUEUE_NAME is the name of the local queue.

13. For each cloud queue, verify that the value of resources_default.node_location is unset.

14. Restart or HUP the scheduler:kill -HUP <scheduler PID>

Proprietary Information of Altair Engineering

Page 238: Control Administrator's Guide - Altair PBS Works · Altair Control 2019.1 Administrator's Guide iv • Altair Embed SE™ ©1989-2019 (formerly solidThinking Embed® SE) • Altair

Control Administrator's Guide9 Cloud Bursting p.233

9.18 Override Instance Type or Image at JobSubmissionOverride the default instance type or OS image used to burst the cloud node at job submission.

The OS image that is to be used when a cloud node is burst can be specified at job submission via theqsub command. If this information is not provided at job submission, then the OS image defaults to thevalue of the cloud_default_image parameter in the cloud bursting hook configuration file.

Additionally, the instance type that is burst can be specified at job submission via the qsub command(ex. Standard_DS3_v2). Instance types are defined by the cloud provider. The name of the instancetype must exactly match the cloud provider's. If this information is not provided at job submission, thenthe instance type defaults to instance type defined for the cloud queue.

1. Override the cloud image at job submission via the qsub command:qsub -l select=1:ncpus=4 -q QUEUE_NAME -v CLOUD_IMAGE=IMAGE JOB_SCRIPT

Where:

• QUEUE_NAME is the name of the cloud queue.

• JOB_SCRIPT is the name of the script to be executed.

• IMAGE is the name of the OS image to be used when the node is burst.

2. Override the cloud instance at job submission via the qsub command:qsub -l select=1:ncpus=4 -q QUEUE_NAME -v CLOUD_INSTANCE=INSTANCE_TYPE JOB_SCRIPT

Where:

• QUEUE_NAME is the name of the cloud queue.

• JOB_SCRIPT is the name of the script to be executed.

• INSTANCE_TYPE is the name of cloud provider instance type. This is the type of machine thatwill be burst.

3. You can also override both the OS image and the instance type as part of the qsub command:qsub -l select=1:ncpus=4 -q QUEUE_NAME \-v CLOUD_IMAGE=IMAGE,CLOUD_INSTANCE=INSTANCE_TYPE JOB_SCRIPT

Proprietary Information of Altair Engineering

Page 239: Control Administrator's Guide - Altair PBS Works · Altair Control 2019.1 Administrator's Guide iv • Altair Embed SE™ ©1989-2019 (formerly solidThinking Embed® SE) • Altair

Control Administrator's Guide9 Cloud Bursting p.234

9.19 Request InfiniBand NodesRequest InfiniBand nodes to run HPC workloads like MPI jobs.

You will need to create an InfiniBand enabled OS image before you can submit a job to an InfiniBandcloud node.

When the user wants to run HPC workloads like MPI jobs, the user needs InfiniBand supported nodesdeployed on the same InfiniBand network. At job submission, the user can request InfiniBand nodes byrequesting InfiniBand supported instance types, an InfiniBand enabled OS image, and can request thesenodes to be on the same InfiniBand network using the option -vCLOUD_NETWORK=IB.

Among the cloud providers, only Azure supports InfiniBand nodes today.

Request Infiniband nodes by overriding the cloud image and the cloud instance via the qsub command:qsub -l select=1:ncpus=4 -q QUEUE_NAME \-v CLOUD_IMAGE=IMAGE,CLOUD_INSTANCE=INSTANCE_TYPE,CLOUD_NETWORK=IB JOB_SCRIPT

Where:

• QUEUE_NAME is the name of the cloud queue.

• INSTANCE_TYPE is the name of cloud provider instance type (machine types, shapes or flavors)and should be an instance type that supports Infiniband.

• IMAGE is the name of the Infiniband enabled OS image to be used when the node is burst.

• JOB_SCRIPT is the name of the script to be executed.

Proprietary Information of Altair Engineering

Page 240: Control Administrator's Guide - Altair PBS Works · Altair Control 2019.1 Administrator's Guide iv • Altair Embed SE™ ©1989-2019 (formerly solidThinking Embed® SE) • Altair

Control Administrator's Guide9 Cloud Bursting p.235

9.20 AWS Spot PricingUse AWS Spot Instances to request spare computing capacity for up to 90% off the On-Demand price.

9.20.1 AWS Spot Pricing OverviewA description of spot pricing and information on implementing it.

Note: Spot pricing is only available with Control 2019.1 and PBS Professional 19.2.1.

What is Spot PricingAmazon EC2 Spot Instances offer spare compute capacity available in the AWS cloud at steep discountscompared to On-Demand instances. Spot Instances can be interrupted by AWS with two minutes ofnotification when AWS needs the capacity back. AWS can interrupt your Spot Instance when the Spotprice exceeds your maximum price, when the demand for Spot Instances rises, or when the supply ofSpot Instances decreases.

The recommendation is to avoid running critical or long running jobs on spot instances as jobs may bekilled when spot instances are preempted.

Implementing AWS Spot PricingCreate an AWS bursting scenario. Once the bursting scenario is created, select the bursting scenarioand choose to setup spot instances. A list of machine types are displayed that can be enabled for spotpricing:

Figure 83: Spot Instances

Proprietary Information of Altair Engineering

Page 241: Control Administrator's Guide - Altair PBS Works · Altair Control 2019.1 Administrator's Guide iv • Altair Embed SE™ ©1989-2019 (formerly solidThinking Embed® SE) • Altair

Control Administrator's Guide9 Cloud Bursting p.236

Instance type nameName of the instance type (machine type, shapes or flavors) as defined by the cloud provider.

CoreNumber of CPUs available for this instance type.

MemAmount of RAM available for this instance type.

GPUNumber of GPUs available for this instance type.

PriceOn-demand price. With On-Demand instances, you pay for compute capacity by per hour or persecond depending on which instances you run. No longer-term commitments or upfront paymentsare needed. You can increase or decrease your compute capacity depending on the demands ofyour application and only pay the specified per hourly rates for the instance you use.

Current Spot PriceCurrently, available spot price. With spot instances, you pay the spot price that's in effect forthe time period your instances are running. Spot instance prices are set by Amazon EC2 andadjust gradually based on long-term trends in supply and demand for Spot instance capacity. Spotinstances allow you to request spare Amazon EC2 computing capacity for up to 90% off the On-Demand price.

Max Spot PriceMaximum amount that your site is willing to pay to continue using the spot instance. Once thisprice is exceeded the spot instance can be interrupted by AWS with two minutes of notification.

Enable for spotMachine type is enabled for spot pricing.

Choose which instance types that you want for spot pricing and enter a maximum price that you arewilling to pay to continue using the spot instance.

A scenario must be added to the cloud bursting hook configuration file for every instance type selectedfor spot pricing and the scenario must be enabled as preemptible.

"preemptable": true

Proprietary Information of Altair Engineering

Page 242: Control Administrator's Guide - Altair PBS Works · Altair Control 2019.1 Administrator's Guide iv • Altair Embed SE™ ©1989-2019 (formerly solidThinking Embed® SE) • Altair

Control Administrator's Guide9 Cloud Bursting p.237

Cloud Bursting Hook Scenario Example

Two instance types are selected for spot pricing:

Figure 84: Spot Instances

Add the following scenario to the cloud bursting hook configuration file:

"aws_scenario_1": { "api_key": "API-KEY-HERE", "cloud_default_image": "<CloudDefaultImage>", "instance_types":{ "a1.2xlarge": {"ncpus": 8, "mem": "16gb" }, "a1.medium": {"ncpus": 1, "mem": "2gb"} }, "cloud_max_instances": 20, "max_nodes_per_burst":50, "cloud_node_wait_timeout":180 "preemptable": true }

Or, you can set up separate scenarios for each instance type:

"aws_scenario_1": { "api_key": "API-KEY-HERE", "cloud_default_image": "<CloudDefaultImage>", "instance_types":{ "a1.2xlarge": {"ncpus": 8, "mem": "16gb" }, }, "cloud_max_instances": 20, "max_nodes_per_burst":50, "cloud_node_wait_timeout":180 "preemptable": true },"aws_scenario_2": { "api_key": "API-KEY-HERE", "cloud_default_image": "<CloudDefaultImage>", "instance_types":{ "a1.medium": {"ncpus": 1, "mem": "2gb"} }, "cloud_max_instances": 20, "max_nodes_per_burst":50, "cloud_node_wait_timeout":180 "preemptable": true }

Proprietary Information of Altair Engineering

Page 243: Control Administrator's Guide - Altair PBS Works · Altair Control 2019.1 Administrator's Guide iv • Altair Embed SE™ ©1989-2019 (formerly solidThinking Embed® SE) • Altair

Control Administrator's Guide9 Cloud Bursting p.238

9.20.2 Choose Spot InstancesChoose instance types for spot pricing and set the maximum spot price.

Before you can choose instances for spot pricing you must have created an AWS bursting scenario.

1. Login to Control.

2. Click the Cloud tab.

3. Under Infrastructure, click Bursting.

4. Select an AWS bursting scenario by clicking on its name.

5. Choose one of the following options:.

• If spot pricing has not yet been setup, click Setup spot instances.

• If spot pricing instances have already been enabled, click Edit spot instances.

A list of spot instances are listed.

Figure 85: Spot Instances

6. Scroll through the list by:

• Clicking the Previous and Next buttons to scroll through the list.

• Entering a page number in the Page field and pressing ENTER to jump to a specific page.

• Entering search criteria in the Search by name field to filter the list of instance types byname.

7. Select an instance type for spot pricing by enabling Enable for spot to the far right of theinstance type name.

Tip: To disable spot pricing for an instance type, disable Enable for spot.

8. Enter a maximum spot price that you are willing to pay for the instance type.

9. Repeat steps 6 through 8 to choose additional instance types for spot pricing.

10. Click Save.

Proprietary Information of Altair Engineering

Page 244: Control Administrator's Guide - Altair PBS Works · Altair Control 2019.1 Administrator's Guide iv • Altair Embed SE™ ©1989-2019 (formerly solidThinking Embed® SE) • Altair

Control Administrator's Guide9 Cloud Bursting p.239

9.20.3 Add Scenarios to the Cloud Bursting HookAdd scenarios to the cloud bursting hook configuration file to configure PBS Professional to use spotinstances.

Configuring the hook must be done as root or as a user with sudo permissions using the sudocommand.

1. Export the cloud bursting hook configuration to a file:qmgr -c "export hook cloud_hook application/x-config default" > FILENAME

Where FILENAME is the name of the file where the hook configuration is written.

qmgr -c "export hook cloud_hook application/x-config default" > config.json

2. Edit the file.

3. Add the AWS scenario to the scenario container:

Tip: You can create a single scenario with multiple spot instance types or create ascenario for each spot instance type.

a) Set the value of api_key to the token generated for the bursting scenario.

b) Set the value of cloud_default_image to OS image that should be used for bursting.

This OS image is used if the image is not requested as part of the job submission request.

c) For the instance_types container, add a comma separated list of the instance types thathave been enabled for spot pricing."instance_types":{ "a1.2xlarge": {"ncpus": 8, "mem": "16gb" }, "a1.medium": {"ncpus": 1, "mem": "2gb"} },

d) Set the value of cloud_max_instances to the maximum number of instances that can bemade available in the cloud.

Must be expressed an integer greater than zero. This value is overridden by the resourcevalue set at the server-level.

e) Set the value of max_nodes_per_burst to the maximum number of nodes allowed to burst ina single hook cycle.

f) Set the value of cloud_node_wait_timeout to the maximum time to wait for freshly burstnode to become usable.

Must be expressed as an integer whose units are in seconds. Default is 180 seconds. You canset this to a higher value, but a lower value defaults to 180 seconds.

g) Set preemptable to True.

4. Save the file.

5. Re-import the hook with the updated configuration values:qmgr -c "import hook cloud_hook application/x-config default FILENAME"

Where FILENAME is the name of the file containing the hook configuration.qmgr -c "import hook cloud_hook application/x-config default config.json"

6. Enable the cloud bursting hook.

Proprietary Information of Altair Engineering

Page 245: Control Administrator's Guide - Altair PBS Works · Altair Control 2019.1 Administrator's Guide iv • Altair Embed SE™ ©1989-2019 (formerly solidThinking Embed® SE) • Altair

Control Administrator's Guide9 Cloud Bursting p.240

The cloud bursting server periodic hook is disabled by default.qmgr -c "set hook cloud_hook enabled=True"

Proprietary Information of Altair Engineering

Page 246: Control Administrator's Guide - Altair PBS Works · Altair Control 2019.1 Administrator's Guide iv • Altair Embed SE™ ©1989-2019 (formerly solidThinking Embed® SE) • Altair

Control Administrator's Guide9 Cloud Bursting p.241

9.21 Modify the Bursting HookAdd, remove or change a scenario in the cloud bursting hook configuration file.

Modifying any of the following parameters in the cloud bursting hook configuration file does not requireany special processing. The cloud bursting hook will use the updated values in the next hook cycle:

• idle_before_unburst

• cloud_min_instances

• resources

9.21.1 Change a Scenario in the Cloud Bursting HookMake modifications to an existing scenario in the cloud bursting hook.

Configuring the hook must be done as root or as a user with sudo permissions using the sudocommand.

1. Login to the PBS Server.

2. Export the cloud bursting hook configuration to a file:qmgr -c "export hook cloud_hook application/x-config default" > FILENAME

Where FILENAME is the name of the file where the hook configuration is written.

qmgr -c "export hook cloud_hook application/x-config default" > config.json

3. Edit the file.

4. Make any necessary adjustments to the scenario.

5. Save the file.

6. Re-import the hook with the updated configuration values:qmgr -c "import hook cloud_hook application/x-config default FILENAME"

Where FILENAME is the name of the file containing the hook configuration.

9.21.2 Add a New Scenario to the Cloud Bursting HookModify the cloud bursting hook configuration file by adding a new scenario.

Configuring the hook must be done as root or as a user with sudo permissions using the sudocommand.

1. Login to the PBS Server.

2. Export the cloud bursting hook configuration to a file:qmgr -c "export hook cloud_hook application/x-config default" > FILENAME

Where FILENAME is the name of the file where the hook configuration is written.

qmgr -c "export hook cloud_hook application/x-config default" > config.json

Proprietary Information of Altair Engineering

Page 247: Control Administrator's Guide - Altair PBS Works · Altair Control 2019.1 Administrator's Guide iv • Altair Embed SE™ ©1989-2019 (formerly solidThinking Embed® SE) • Altair

Control Administrator's Guide9 Cloud Bursting p.242

3. Edit the file.

4. Add a scenario by adding the following information to the scenario container:

Note: Scenarios are specific to a cloud provider.

a) Set the value of api_key to the token generated for the bursting scenario.

b) Set the value of cloud_default_image to OS image that should be used for bursting.

This OS image is used if the image is not requested as part of the job submission request.

c) For the instance_types container, add a comma separated list of any instance types yoursite may want to burst.

Instance types are defined by the cloud provider. The name of the instance type and the CPU,GPU and memory signature must exactly match the cloud provider's."instance_types":{ "t2.large": {"ncpus": 2, "mem": "8gb" }, "t2.xlarge": {"ncpus": 4, "mem": "16gb" }, "t2.2xlarge": {"ncpus": 8, "mem": "32gb"} },

d) Set the value of cloud_max_instances to the maximum number of instances that can bemade available in the cloud.

Is expressed as an integer whose units are in seconds. This value is overridden by theresource value set at the server-level.

e) Optionally, set the value of max_vms_for_infiniband_scaleset to define the maximumnumber of nodes allowed on a single InfiniBand switch.

This value should match the value of Maximum number of VMs inside a ScaleSet as specifiedin the Azure bursting scenario. If you are not using InfiniBand, then you can eliminate thisparameter.

Note: Supported by Azure only.

f) Set the value of max_nodes_per_burst to the maximum number of nodes allowed to burst ina single hook cycle.

g) Set the value of cloud_node_wait_timeout to the maximum time to wait for freshly burstnode to become usable.

Must be expressed as an integer whose units are in seconds. Default is 180 seconds. You canset this to a higher value, but a lower value defaults to 180 seconds.

h) Set preemptable to:

• True when the bursting scenario supports preemptible (spot) instances and cloud jobsmay be preempted.

• False when jobs that are run in the cloud should not be preemptible.

Default value is False for a scenario. If flag is not specified, the default value is used.

5. Save the file.

6. Re-import the hook with the updated configuration values:qmgr -c "import hook cloud_hook application/x-config default FILENAME"

Proprietary Information of Altair Engineering

Page 248: Control Administrator's Guide - Altair PBS Works · Altair Control 2019.1 Administrator's Guide iv • Altair Embed SE™ ©1989-2019 (formerly solidThinking Embed® SE) • Altair

Control Administrator's Guide9 Cloud Bursting p.243

Where FILENAME is the name of the file containing the hook configuration.

Now you must add and configure a new cloud queue for the scenario.

9.21.3 Delete a Scenario from the Cloud Bursting HookRemove a scenario that is no longer needed from the cloud bursting hook configuration file.

Before deleting a scenario from the cloud bursting hook configuration file, disable cloud bursting.

Configuring the hook must be done as root or as a user with sudo permissions using the sudocommand.

1. Login to the PBS Server.

2. Export the cloud bursting hook configuration to a file:qmgr -c "export hook cloud_hook application/x-config default" > FILENAME

Where FILENAME is the name of the file where the hook configuration is written.

qmgr -c "export hook cloud_hook application/x-config default" > config.json

3. Edit the file.

4. Remove the scenario from the scenario container.

5. Save the file.

6. Re-import the hook with the updated configuration values:qmgr -c "import hook cloud_hook application/x-config default FILENAME"

Where FILENAME is the name of the file containing the hook configuration.

7. Enable all cloud queues.qmgr -c "set queue QUEUE_NAME enabled=true"

Where QUEUE_NAME is the name of the cloud queue.

9.21.4 Change the PCLM ServerUpdate the hook configuration file when the location of the Cloud component changes.

Before making any changes to the endpoint for accessing the Cloud component, disable cloud bursting.

1. Login to the PBS Server.

2. Export the cloud bursting hook configuration to a file:qmgr -c "export hook cloud_hook application/x-config default" > FILENAME

Where FILENAME is the name of the file where the hook configuration is written.

qmgr -c "export hook cloud_hook application/x-config default" > config.json

3. Edit the file.

4. Set pclm_server to the endpoint for accessing Cloud in the format http://HOSTNAME:PORT.

Proprietary Information of Altair Engineering

Page 249: Control Administrator's Guide - Altair PBS Works · Altair Control 2019.1 Administrator's Guide iv • Altair Embed SE™ ©1989-2019 (formerly solidThinking Embed® SE) • Altair

Control Administrator's Guide9 Cloud Bursting p.244

Where HOSTNAME is the IP address or hostname of the machine where Cloud is installed andPORT is the Cloud port (Defaults to 9980).

5. Save the file.

6. Re-import the hook with the updated configuration values:qmgr -c "import hook cloud_hook application/x-config default FILENAME"

Where FILENAME is the name of the file containing the hook configuration.

7. Enable all cloud queues.qmgr -c "set queue QUEUE_NAME enabled=true"

Where QUEUE_NAME is the name of the cloud queue.

Proprietary Information of Altair Engineering

Page 250: Control Administrator's Guide - Altair PBS Works · Altair Control 2019.1 Administrator's Guide iv • Altair Embed SE™ ©1989-2019 (formerly solidThinking Embed® SE) • Altair

Control Administrator's Guide9 Cloud Bursting p.245

9.22 Disable Cloud BurstingDisable cloud bursting by disabling and draining the cloud queues, waiting until the cloud nodesunburst, and disabling the cloud bursting hook.

Disabling cloud bursting must be done as root or as a user with sudo permissions using the sudocommand.

1. Login to the PBS Server.

2. Disable all cloud queues.qmgr -c "set queue QUEUE_NAME enabled=false"

Where QUEUE_NAME is the name of the cloud queue.

3. Drain the cloud queues by allowing time for the jobs that are waiting in the queues to run andcomplete.

Tip: If a running job is taking a long time to complete, it can be deleted using theqdel command.

4. Verify that all cloud nodes are unburst:

a) Login to Control.

b) Click Cloud.

c) Click the Bursting tab on the left-hand side of the web page.

d) Click on the name of a cloud bursting scenario.

e) View the information displayed under the Machines heading.

When the following message is displayed, all cloud nodes are unburst.No machines are available

f) Repeat steps 4.d to 4.e until all cloud nodes are unburst for each cloud bursting scenario.

5. Disable the cloud bursting hook.qmgr -c "set hook cloud_hook enabled=False"

Proprietary Information of Altair Engineering

Page 251: Control Administrator's Guide - Altair PBS Works · Altair Control 2019.1 Administrator's Guide iv • Altair Embed SE™ ©1989-2019 (formerly solidThinking Embed® SE) • Altair

Control Administrator's Guide9 Cloud Bursting p.246

9.23 Uninstall PCLM CLI and Cloud Bursting HookUninstall the PCLM CLI and the cloud bursting hook when it is no longer needed.

1. Login to the PBS Server.

2. Navigate to where the contents of the pbspro_cloud_bursting_hook.tar.gz tarball file wereextracted.

3. Navigate to the directory containing install.sh.

4. Uninstall the PCLM CLI and the cloud bursting hook:./install.sh --type uninstall

Proprietary Information of Altair Engineering

Page 252: Control Administrator's Guide - Altair PBS Works · Altair Control 2019.1 Administrator's Guide iv • Altair Embed SE™ ©1989-2019 (formerly solidThinking Embed® SE) • Altair

Upgrade or Reconfigure PBSProfessional 10

10 Upgrade or Reconfigure PBS Professional

Instructions for how to handle upgrading or reconfiguring PBS Professional.

This chapter covers the following:

• 10.1 Prepare for a PBS Professional Upgrade (p. 248)

• 10.2 Reconfigure Bursting After a PBS Professional Upgrade (p. 249)

• 10.3 Start the Data Collectors (p. 251)

Page 253: Control Administrator's Guide - Altair PBS Works · Altair Control 2019.1 Administrator's Guide iv • Altair Embed SE™ ©1989-2019 (formerly solidThinking Embed® SE) • Altair

Control Administrator's Guide10 Upgrade or Reconfigure PBS Professional p.248

10.1 Prepare for a PBS Professional UpgradeSave the cloud bursting hook configuration file and stop the Data Collectors before upgrading PBSProfessional.

Before upgrading PBS Professional, performing maintenance, or re-configuring execution hosts wherethe system is drained of running jobs and the PBS Professional Server and MoMs are shut down, it isrecommended to first disable cloud bursting.

If you site has implemented cloud bursting or has installed any Data Collectors, save the cloud burstinghook configuration file and stop any Data Collectors.

The following steps must be done as root or as a user with sudo permissions using the sudo command.

1. Login to the PBS Server.

2. Export the cloud bursting hook configuration to a file:qmgr -c "export hook cloud_hook application/x-config default" > FILENAME

Where FILENAME is the name of the file where the hook configuration is written.

3. Save the cloud bursting hook configuration file to a secure location.

4. Login to each machine where a Data Collector is installed.

5. Stop the Data Collector:pbs-control-manager stop

Now you may upgrade PBS Professional using the instructions in the chapter Upgrading in the PBSProfessional Installation & Upgrade Guide.

Proprietary Information of Altair Engineering

Page 254: Control Administrator's Guide - Altair PBS Works · Altair Control 2019.1 Administrator's Guide iv • Altair Embed SE™ ©1989-2019 (formerly solidThinking Embed® SE) • Altair

Control Administrator's Guide10 Upgrade or Reconfigure PBS Professional p.249

10.2 Reconfigure Bursting After a PBS ProfessionalUpgradeReconfigure PBS Professional for cloud bursting after an upgrade.

10.2.1 Create an OS Image for the New PBS MoMCreate a virtual machine, install and configure the new PBS MoM, and create an OS image of the virtualmachine.

Use the instructions for creating a virtual machine, installing the PBS MoM and creating an OS image asyou did when you first configured PBS Professional for cloud bursting.

Repeat the following steps for each cloud provider that your site is using for cloud bursting.

1. Login to your cloud provider account.

2. Create a new virtual machine.

3. Install and configure the PBS MoM on the virtual machine.

4. Create an OS image of the virtual machine.

10.2.2 Update Bursting Scenarios with the New ImageUpdate all existing bursting scenarios with the new image of the PBS MoM.

1. Login to Control.

2. Click the Cloud tab.

3. Under Infrastructure, click Bursting.

4. Select a bursting scenario by clicking on its name.

5. Click .

Figure 86: Edit Bursting Scenario

6. Update the OS image with the appropriate information based on the cloud provider:

To obtain the information for updating the bursting scenario's OS image see the topic for creatingthe original bursting scenario. A brief description is provided below:

Proprietary Information of Altair Engineering

Page 255: Control Administrator's Guide - Altair PBS Works · Altair Control 2019.1 Administrator's Guide iv • Altair Embed SE™ ©1989-2019 (formerly solidThinking Embed® SE) • Altair

Control Administrator's Guide10 Upgrade or Reconfigure PBS Professional p.250

Cloud Provider Information to Enter for OS Image

AWS Amazon Machine Image (AMI) ID

Azure Using managed disks, the name of the image. Otherwise, the image'sLinux Source BLOB URI.

GCP The value for the entry called selfLink in the image's REST responsefrom the Google Compute Engine service.

Oracle Image OCID

Orange FlexibleEngine

Image ID

HUAWEI Image ID

OTC Image ID

7. Repeat steps 4 through 6 for each bursting scenario.

See AlsoCreate an AWS Cloud Bursting ScenarioCreate an Azure Cloud Bursting ScenarioCreate a GCP Cloud Bursting ScenarioCreate an Oracle Cloud Bursting ScenarioCreate a Orange Cloud Cloud Bursting ScenarioCreate an HUAWEI Cloud Bursting ScenarioCreate an OTC Cloud Bursting Scenario

10.2.3 Configure PBS Professional for Cloud BurstingConfigure PBS Professional 19.2.1 for cloud bursting.

See Configure PBS Professional for Cloud Bursting for instructions on how to create the necessarycomponents for cloud bursting. The PBS Professional 19.2.1 cloud bursting hook and the structure ofits configuration file have changed to support poly-cloud bursting and AWS spot instances. The cloudbursting hook configuration file from previous versions of PBS Professional can be used as a startingpoint for configuring the 19.2.1 cloud bursting hook, but cannot be imported as is.

Proprietary Information of Altair Engineering

Page 256: Control Administrator's Guide - Altair PBS Works · Altair Control 2019.1 Administrator's Guide iv • Altair Embed SE™ ©1989-2019 (formerly solidThinking Embed® SE) • Altair

Control Administrator's Guide10 Upgrade or Reconfigure PBS Professional p.251

10.3 Start the Data CollectorsStart the Data Collectors so that accounting log information is transported to Control.

1. Login to each machine where a Data Collector is installed.

2. Start the Data Collector:pbs-control-manager start

Proprietary Information of Altair Engineering

Page 257: Control Administrator's Guide - Altair PBS Works · Altair Control 2019.1 Administrator's Guide iv • Altair Embed SE™ ©1989-2019 (formerly solidThinking Embed® SE) • Altair

Advanced Configuration 11

11 Advanced Configuration

This chapter covers the following:

• 11.1 Advanced Configuration by Component (p. 253)

• 11.2 Advanced Configuration for the Web Interface (p. 255)

• 11.3 Advanced Configuration for Monitoring (p. 263)

• 11.4 Advanced Configuration for Analyze (p. 266)

• 11.5 Advanced Configuration for Cloud (p. 269)

Page 258: Control Administrator's Guide - Altair PBS Works · Altair Control 2019.1 Administrator's Guide iv • Altair Embed SE™ ©1989-2019 (formerly solidThinking Embed® SE) • Altair

Control Administrator's Guide11 Advanced Configuration p.253

11.1 Advanced Configuration by ComponentOverview of advanced configuration that is available for each Control component and associatedconfiguration files.

Web Interface

Table 14: Advanced Configuration and Associated Files for the Web Interface

Configuration Category Configuration File

Single Sign-on Authentication PC_HOME/config/shared/auth-config/auth_details.json

PC_HOME/config/shared/auth-config/krb5.conf

PC_HOME/config/pcgui/tomcat/conf/web.xml

Multi-Factor Authentication PC_HOME/config/shared/auth-config/auth_details.json

SSH Server PC_HOME/config/ams/jaas-config/AA_jaas.config

Session Timeout PC_HOME/config/ams/AA_Service.properties

Gateway HTTPS Port PC_HOME/config/api_gateway/nginx/nginx.conf

License Server PC_HOME/config/license/environment.json

PC_HOME/pbs-control-simulator/etc/pbssim/environment

PC_EXEC/pbsa/scripts/server-start.sh

Webhelp PC_HOME/config/api_gateway/nginx/conf.d/pc.conf

Proprietary Information of Altair Engineering

Page 259: Control Administrator's Guide - Altair PBS Works · Altair Control 2019.1 Administrator's Guide iv • Altair Embed SE™ ©1989-2019 (formerly solidThinking Embed® SE) • Altair

Control Administrator's Guide11 Advanced Configuration p.254

Analyze

Table 15: Advanced Configuration and Associated Files for Analyze

Configuration Parameter Configuration File

Location of Accounting Logs PC_HOME/config/dc/modules/pbsa/config/config.xml

Analyze Hostname PC_HOME/config/api_gateway/nginx/nginx.conf

PC_HOME/pbs-control-simulator/etc/pbssim/environment

PC_HOME/pbs-control-simulator/etc/pbssim/pbssim.conf

Monitor

Table 16: Advanced Configuration and Associated Files for Monitor

Configuration Parameter Configuration File

Concurrent Threads for Retrieving JobDetails

PC_HOME/config/monitor/services/monitor.conf

Job and Node Details Refresh Interval PC_HOME/config/monitor/services/monitor.conf

Session Timeout PC_HOME/config/monitor/services/monitor.conf

Channel Timeout PC_HOME/config/monitor/services/monitor.conf

Cloud

Table 17: Advanced Configuration and Associated Files for Cloud

Configuration Parameter Configuration File

Cloud Hostname PC_HOME/config/api_gateway/nginx/nginx.conf

PC_HOME/config/api_gateway/nginx/conf.d/model.conf

PC_HOME/config/shared/token_handler_config/token_validator/RequestValidatorConfig.json

DNS Nameservers /etc/docker/daemon.json

Proprietary Information of Altair Engineering

Page 260: Control Administrator's Guide - Altair PBS Works · Altair Control 2019.1 Administrator's Guide iv • Altair Embed SE™ ©1989-2019 (formerly solidThinking Embed® SE) • Altair

Control Administrator's Guide11 Advanced Configuration p.255

11.2 Advanced Configuration for the Web Interface

11.2.1 Advanced Configuration for Authentication

Configure Single Sign-On

Enable Single Sign-OnEnable Single Sign-On (SSO) so that clients who are logged into a domain can access Control withoutproviding credentials a second time.

The browser that is used to access Control has to be configured for Kerberos and NT LAN Manager(NTLM). View How to configure supported browsers for Kerberos and NTLM for a list of supportedbrowsers and platforms, as well as steps for configuring the supported browser.

Additionally, every service that uses Kerberos authentication needs to have an SPN set for it sothat clients can identify the service on the network. For information on how to register a SPN seeRegister a SPN. If a SPN has already been registered, verify the SPN before enabling SSO - Verify SPNRegistrations.

The following steps must be done as root or as a user with sudo permissions using the sudo command.

Note: By default SSO is disabled.

1. Login to the machine where the Web Interface is installed.

2. Edit the file PC_HOME/config/shared/auth-config/auth_details.json.

3. Enable SSO by changing the value of the sso key to true."sso": { "enable_sso": "true" },

4. Edit the file PC_HOME/config/shared/auth-config/krb5.conf.

5. Replace all instances of $SSO_DOMAIN$ with the domain name (in uppercase) against which theusers are authenticated.[libdefaults] default_realm=$SSO_DOMAIN$ default_tkt_enctypes = aes128-cts rc4-hmac des3-cbc-sha1 des-cbc-md5 des-cbc-crc default_tgs_enctypes = aes128-cts rc4-hmac des3-cbc-sha1 des-cbc-md5 des-cbc-crc permitted_enctypes = aes128-cts rc4-hmac des3-cbc-sha1 des-cbc-md5 des-cbc-crc

[realms] $SSO_DOMAIN$ = { kdc = $SSO_DOMAIN$ default_domain = $SSO_DOMAIN$}

[domain_realm] .$SSO_DOMAIN$ = $SSO_DOMAIN$

On Windows, find the domain name by right-clicking My Computer and selecting Properties.

Proprietary Information of Altair Engineering

Page 261: Control Administrator's Guide - Altair PBS Works · Altair Control 2019.1 Administrator's Guide iv • Altair Embed SE™ ©1989-2019 (formerly solidThinking Embed® SE) • Altair

Control Administrator's Guide11 Advanced Configuration p.256

Figure 87: Windows Domain Name

On Linux, find the domain name using the command dnsdomainsname.

6. Edit the file PC_HOME/config/pcgui/tomcat/conf/web.xml.

7. Perform a search for the following text: ALTAIR PBSControl SSO Configurations.

8. Uncomment the XML just below this text by removing <!-- and --> that surround the XML.<!--<filter> <filter-name>SpnegoHttpFilter</filter-name> <filter-class>net.sourceforge.spnego.SpnegoHttpFilter</filter-class>

<init-param> <param-name>spnego.allow.basic</param-name> <param-value>true</param-value> </init-param>

<init-param> <param-name>spnego.allow.localhost</param-name> <param-value>true</param-value> </init-param>….…. <init-param> <param-name>spnego.logger.level</param-name> <param-value>1</param-value> </init-param></filter>

<filter-mapping> <filter-name>SpnegoHttpFilter</filter-name> <url-pattern>*.html</url-pattern></filter-mapping>-->

9. Update SPN_USER_NAME and SPN_USER_PASSWORD with the credentials used when registeringthe SPN. <init-param> <param-name>spnego.preauth.username</param-name> <param-value>{SPN_USER_NAME}</param-value> </init-param>

<init-param> <param-name>spnego.preauth.password</param-name> <param-value>{SPN_USER_PASSWORD}</param-value> </init-param>

10. Restart Control:pbs-control-manager restart

Proprietary Information of Altair Engineering

Page 262: Control Administrator's Guide - Altair PBS Works · Altair Control 2019.1 Administrator's Guide iv • Altair Embed SE™ ©1989-2019 (formerly solidThinking Embed® SE) • Altair

Control Administrator's Guide11 Advanced Configuration p.257

Register a SPNRegister a Service Principal Name (SPN) so the you can implement Kerberos authentication.

SPNs are unique identifiers for services running on servers. Every service that uses Kerberosauthentication needs to have an SPN set for it so that clients can identify the service on the network.

1. Click this link: http://spnego.sourceforge.net/spnego_tomcat.html.

2. Scroll down until you find the section titled "Registering an SPN".

3. Follow the instructions to register a SPN.

Verify SPN RegistrationsCheck that a SPN is registered for a specific computer.

1. Click the Windows Start button.

2. Enter command in the search box.

3. Right-click Command Prompt.4. Click Run as administrator.5. Enter the command: setspn -T -F -Q http/<DNSNAME>.

<DNSNAME> is the DNS hostname used when registering the SPN.

Disable Single Sign-OnDisable Single Sign-On so that users are forced to enter their credentials when logging into Control.

The following steps must be done as root or as a user with sudo permissions using the sudo command.

1. Login to the machine where the Web Interface is installed.

2. Edit the file PC_HOME/config/shared/auth-config/auth_details.json.

3. Disable SSO by changing the value of the sso key to false."sso": { "enable_sso": "false" },

4. Edit the file PC_HOME/config/pcgui/tomcat/conf/web.xml.

5. Perform a search for the following text: ALTAIR PBSControl SSO Configurations.

6. Comment the XML just below this text by surrounding the XML with <!-- and -->.<!--<filter> <filter-name>SpnegoHttpFilter</filter-name> <filter-class>net.sourceforge.spnego.SpnegoHttpFilter</filter-class>

<init-param> <param-name>spnego.allow.basic</param-name> <param-value>true</param-value> </init-param>

<init-param> <param-name>spnego.allow.localhost</param-name> <param-value>true</param-value> </init-param>….….

Proprietary Information of Altair Engineering

Page 263: Control Administrator's Guide - Altair PBS Works · Altair Control 2019.1 Administrator's Guide iv • Altair Embed SE™ ©1989-2019 (formerly solidThinking Embed® SE) • Altair

Control Administrator's Guide11 Advanced Configuration p.258

<init-param> <param-name>spnego.logger.level</param-name> <param-value>1</param-value> </init-param></filter>

<filter-mapping> <filter-name>SpnegoHttpFilter</filter-name> <url-pattern>*.html</url-pattern></filter-mapping>-->

7. Restart Control:pbs-control-manager restart

Enable Multi-Factor AuthenticationEnable DUO multi-factor authentication to strengthen access security by requiring two methods to verifyyour identity.

You must have already implemented DUO multi-factor authentication at your site.

The following steps must be done as root or as a user with sudo permissions using the sudo command.

Multi-factor authentication adds a second layer of security to your site. Verifying your identity using asecond factor (like your phone or other mobile device) prevents anyone but you from logging in, even ifthey know your password.

1. Login to the machine where the Web Interface is installed.

2. Edit the file PC_HOME/config/shared/auth-config/auth_details.json.

3. Enable multi-factor authentication by changing the value of the enable_multi_factor key to true. "multifactor": { "enable_multi_factor": "true",

4. Replace %INTEGRATION_KEY%, %SECRET_KEY%, and %APPLICATION_KEY% with theappropriate values obtained when you chose to protect a new application using the DUO AdminPanel."integration_key":"%INTEGRATION_KEY%","secret_key":"%SECRET_KEY%","application_key":"%APPLICATION_KEY%",

5. Replace %DUO_SCREEN_URL% with the URL that is hosting DUO authentication.

Note: Control sends requests to this URL and communication must be done overHTTPS - i.e. https://someURL. Additionally, the request that Control sends is a signedrequest, therefore you must be prepared to accept a signed request. For an example,see the HTML in step 5.

"duo_screen_URL":"%DUO_SCREEN_URL%"

6. Make sure your post_action in the DUO web page points to the following URL:https://PC_HOSTNAME/auth/validate/mf

where PC_HOSTNAME is the hostname of the machine where Control is installed.

Example of DUO HTML page:<!DOCTYPE html>

Proprietary Information of Altair Engineering

Page 264: Control Administrator's Guide - Altair PBS Works · Altair Control 2019.1 Administrator's Guide iv • Altair Embed SE™ ©1989-2019 (formerly solidThinking Embed® SE) • Altair

Control Administrator's Guide11 Advanced Configuration p.259

<html> <head> <meta charset="utf-8"> <title>Duo Security 2-factor auth Tut - Enable Two-Factor Auth</title> <link rel="stylesheet" href="css/main.css"> </head><body> <iframe id="duo_iframe" width="100%" height="500" frameborder="0"></iframe> <script src="js/Duo-Web-v2.js" type="text/javascript"></script> <!-- The Duo IFRAME ---> <!-- <iframe id="duo_iframe" width="100%" height="500" frameborder="0"></iframe> --> <!-- <script src="js/Duo-Web-v2.js" type="text/javascript"></script> --> <script> var urlParams = new URLSearchParams(window.location.search); var myVar = urlParams.get('sig_request'); console.log(urlParams.get('sig_request')); console.log(urlParams.toString()); Duo.init({ 'host': 'api-45d973a0.duosecurity.com', 'sig_request': myVar, 'post_action': 'https://PC_HOSTNAME/auth/validate/mf' }); </script> Welcome to DUO login</body></html>

Change the SSH ServerPoint to a new SSH server.

The following steps must be done as root or as a user with sudo permissions using the sudo command.

1. Login to the machine where the Web Interface is installed.

2. Edit the file PC_HOME/config/ams/jaas-config/AA_jaas.config.

3. Change the value of host to the hostname of the new SSH server.LoginModule { com.altair.jaas.module.SSHUnixLoginModule Required host="localhost" port="22" cache_ttl="60" debug=false; com.altair.jaas.module.UsersAccessControl Required;};

4. Restart Control:pbs-control-manager restart

Proprietary Information of Altair Engineering

Page 265: Control Administrator's Guide - Altair PBS Works · Altair Control 2019.1 Administrator's Guide iv • Altair Embed SE™ ©1989-2019 (formerly solidThinking Embed® SE) • Altair

Control Administrator's Guide11 Advanced Configuration p.260

11.2.2 Advanced Configuration for Licensing

Configure the License ServerChange the location of the license server.

The following steps must be done as root or as a user with sudo permissions using the sudo command.

1. Login to the machine hosting the Web Interface.

2. Back up the file PC_HOME/config/license/environment.json.

3. Edit the file environment.json.

4. Update the value of ALTAIR_LICENSE_PATH to the new license server in the formatport@hostname."ALTAIR_LICENSE_PATH":"6200@dm-10",

5. Save the file.

6. Restart Control:pbs-control-manager restart

7. Login to the machine hosting the Simulator.

8. Back up the file PC_HOME/pbs-control-simulator/etc/pbssim/environment.

9. Edit the file environment.

10. Update the value of ALTAIR_LICENSE_PATH to the new license server in the [email protected]_LICENSE_PATH=6200@localhost

11. Save the file.

12. Update the PBS Professional license server.qmgr -c 's s pbs_license_info=LIC_PATH'

Where LIC_PATH is the new license server in the format port@hostname.

13. Restart PBS Professional.service pbs restart

14. Restart Control:pbs-control-manager restart

15. Login to the machine hosting Analyze.

16. Back up the file PC_EXEC/pbsa/scripts/server-start.sh.

17. Edit the file server-start.sh.

18. Update the value of LICENSE_SERVER_PATH to the new license server in the [email protected]_OPTS="$JAVA_OPTS -DLICENSE_SERVER_PATH=6200@dm-10"

19. Save the file.

20. Restart Control.pbs-control-manager restart

Proprietary Information of Altair Engineering

Page 266: Control Administrator's Guide - Altair PBS Works · Altair Control 2019.1 Administrator's Guide iv • Altair Embed SE™ ©1989-2019 (formerly solidThinking Embed® SE) • Altair

Control Administrator's Guide11 Advanced Configuration p.261

11.2.3 Configure the Session TimeoutChange when a session closes as a result of inactivity over the client connection.

The following steps must be done as root or as a user with sudo permissions using the sudo command.

The default session timeout value is 5 hours.

1. Login to the machine where the Web Interface is installed.

2. Edit the file PC_HOME/config/ams/AA_Service.properties.

3. Change the value of java.sessionTimeout to a new value in hours.java.sessionTimeout=5

4. Restart Control:pbs-control-manager restart

11.2.4 Change the Gateway HTTPS PortChange the port that the Gateway HTTPS service listen on.

The following steps must be done as root or as a user with sudo permissions using the sudo command.

1. Login to the machine where the Web Interface is installed.

2. Edit the file PC_HOME/config/api_gateway/nginx/nginx.conf.

3. Change the Gateway HTTPS default port to a new port by updating the default value of 9500 oflisten of server. server { listen 9500 ssl default_server;

server_name _; ... ... }

4. Restart Control:pbs-control-manager restart

11.2.5 Switch to the Locally Installed WebhelpConfigure Control to point to the locally installed webhelp when there is no internet connectivity.

By default, the Control webhelp (User Guide) is served through an Altair documentation portal. Internetconnectivity is necessary to access the portal. You can configure Control to point to webhelp that isinstalled locally when there is no internet connectivity.

1. Login to the machine where the Web Interface is installed.

2. Edit the file PC_HOME/config/api_gateway/nginx/conf.d/pc.conf.

3. Uncomment the below line of the file by removing the "#".#return 302 https://$http_host/pc/docs/index.htm;

4. Comment out the below line of the file by adding a "#" to the beginning of the line:return 302 https://connect.altair.com:443/CP/SA/NA/help/Control/2019.1/index.htm;

Proprietary Information of Altair Engineering

Page 267: Control Administrator's Guide - Altair PBS Works · Altair Control 2019.1 Administrator's Guide iv • Altair Embed SE™ ©1989-2019 (formerly solidThinking Embed® SE) • Altair

Control Administrator's Guide11 Advanced Configuration p.262

The line should look like this to comment it out.#return 302 https://connect.altair.com:443/CP/SA/NA/help/Control/2019.1/index.htm;

5. Restart Control:pbs-control-manager restart

Proprietary Information of Altair Engineering

Page 268: Control Administrator's Guide - Altair PBS Works · Altair Control 2019.1 Administrator's Guide iv • Altair Embed SE™ ©1989-2019 (formerly solidThinking Embed® SE) • Altair

Control Administrator's Guide11 Advanced Configuration p.263

11.3 Advanced Configuration for Monitoring

11.3.1 Configure Concurrent Threads for Retrieving JobDetailsChange the number of concurrent threads that are created to retrieve PBS Professional job details whensystem memory issues arise.

Important: Do not change the number of concurrent threads unless your site isexperiencing system memory issues.

Monitor retrieves job details from PBS Professional using concurrent threads. By default, the number ofthreads created is based on the number of system cores multiplied by four (an 8 core system will create8 * 4 = 32 concurrent threads). This multiplication factor has been well tested and provides optimalresults. If your site is experiencing system memory issues, then the number of concurrent threads canbe reduced to free up memory.

CAUTION: The value of concurrent threads should always be set to a positive (great thanzero) number. Setting the number of concurrent threads to a low value will negatively affectjob refresh time. Assigning a higher value when there is a small number of PBS Professionaljobs will not improve performance.

The following steps must be done as root or as a user with sudo permissions using the sudo command.

1. Login to the machine where the Monitor component is installed.

2. Edit the file PC_HOME/config/monitor/services/monitor.conf.

3. Update the value of QSTAT_THREADS.QSTAT_THREADS=0

4. Restart Control:pbs-control-manager restart

Proprietary Information of Altair Engineering

Page 269: Control Administrator's Guide - Altair PBS Works · Altair Control 2019.1 Administrator's Guide iv • Altair Embed SE™ ©1989-2019 (formerly solidThinking Embed® SE) • Altair

Control Administrator's Guide11 Advanced Configuration p.264

11.3.2 Configure the Job and Node Details RefreshIntervalChange how often Control requests job and node details from PBS Professional.

Important: Do not change the refresh interval unless your site is experiencingperformance issues.

Monitor runs a background process which retrieves job and node detail information from PBSProfessional. By default, this information is updated every 300 seconds (5 minutes). This interval can bemodified when performance issues are being observed. Changing this interval affects:

• How often Dashboard charts are refreshed.

• How often job summary and detail information is refreshed.

• How often node summary and detail information is refreshed.

CAUTION: Setting the refresh interval to a low value may overwhelm PBS Professional withcontinuous requests.

The following steps must be done as root or as a user with sudo permissions using the sudo command.

1. Login to the machine where the Monitor is installed.

2. Edit the file PC_HOME/config/monitor/services/monitor.conf.

3. Update the value of REFRESH_SCHEDULER_WAIT_TIME_IN_SECONDS.REFRESH_SCHEDULER_WAIT_TIME_IN_SECONDS=300

4. Restart Control:pbs-control-manager restart

11.3.3 Configure the Monitor Session TimeoutChange the length of time Monitor will attempt to create an SSH session to a remote HPC cluster whena cluster is added.

When a cluster is added to Monitor, an SSH session is created between the server hosting Monitor andthe remote HPC cluster. If the SSH session cannot be created within the timeframe specified by theMonitor session timeout, an error is displayed to the user. Currently, the default is 60 seconds. Increasethe value of this timeout when network or system response time is affecting Monitor's ability to createan SSH session.

The following steps must be done as root or as a user with sudo permissions using the sudo command.

1. Login to the machine where the Monitor component is installed.

2. Edit the file PC_HOME/config/monitor/services/monitor.conf.

3. Update the value of SESSION_CONNECTION_TIMEOUT in milliseconds.SESSION_CONNECTION_TIMEOUT=60000

4. Restart Control:pbs-control-manager restart

Proprietary Information of Altair Engineering

Page 270: Control Administrator's Guide - Altair PBS Works · Altair Control 2019.1 Administrator's Guide iv • Altair Embed SE™ ©1989-2019 (formerly solidThinking Embed® SE) • Altair

Control Administrator's Guide11 Advanced Configuration p.265

11.3.4 Configure the Monitor Channel TimeoutChange the length of time Monitor will attempt to successfully execute a command over an alreadyexisting SSH session.

Channel timeout refers to the maximum time allowed to successfully execute a command via theMonitor tab over an already established SSH session (i.e., after a cluster has been added). Currently,the default is 30 seconds. Increase the value of this timeout when network or system response time isaffecting Monitor's ability to successfully execute a command over an SSH session.

The following steps must be done as root or as a user with sudo permissions using the sudo command.

1. Login to the machine where the Monitor component is installed.

2. Edit the file PC_HOME/config/monitor/services/monitor.conf.

3. Update the value of CHANNEL_CONNECTION_TIMEOUT in milliseconds.CHANNEL_CONNECTION_TIMEOUT=30000

4. Restart Control:pbs-control-manager restart

Proprietary Information of Altair Engineering

Page 271: Control Administrator's Guide - Altair PBS Works · Altair Control 2019.1 Administrator's Guide iv • Altair Embed SE™ ©1989-2019 (formerly solidThinking Embed® SE) • Altair

Control Administrator's Guide11 Advanced Configuration p.266

11.4 Advanced Configuration for Analyze

11.4.1 Change the Location Where the Data CollectorGathers the Accounting LogsChange the location of the PBS Professional accounting logs without reinstalling the Data Collector.

The following steps must be done as root or as a user with sudo permissions using the sudo command.

The main responsibility of the Data Collector is to make the PBS Professional accounting logs and nodeinformation (pbsnodes -av) accessible to Analtyics. The Data Collector has two functions. It first copiesthe accounting logs and node information from your site’s PBS Professional accounting log location toan intermediate holding area. Secondly, it transports the information stored in the holding area to themachine where Analyze is installed.

You can change the location where the PBS Professional accounting logs are stored without having toreinstall the Data Collector.

1. Login to the machine where the Data Collector is installed.

2. Edit the file PC_HOME/config/dc/modules/pbsa/config/config.xml.

3. Change the value of the XML element <entry> with the attribute @key="accounting-log-dir" tothe new location of the PBS Professional accounting log directory.<entry key="accounting-log-dir">/var/spool/PBS/server_priv/accounting</entry>

4. Restart the Data Collector.pbs-control-manager restart

11.4.2 Change the Analyze HostnameChange the hostname of the Analyze component when it is moved to a new host.

The following steps must be done as root or as a user with sudo permissions using the sudo command.

1. Login to the machine where the Web Interface is installed.

2. Edit the file PC_HOME/config/api_gateway/nginx/nginx.conf.

3. Change the Analyze component hostname by updating the existing value of server for upstreamAnalyticsHost in the format of hostname:port.upstream AnalyticsHost{ server dm-04:9000; }

4. Change the Analyze component hostname by updating the existing value of server for upstreamAnalyticsBAHost in the format of hostname:port.upstream AnalyticsBAHost{ server dm-04:9443; }

5. Save the file.

6. Skip to Step 9 if the Web Interface and Simulator are installed on the same machine.

Proprietary Information of Altair Engineering

Page 272: Control Administrator's Guide - Altair PBS Works · Altair Control 2019.1 Administrator's Guide iv • Altair Embed SE™ ©1989-2019 (formerly solidThinking Embed® SE) • Altair

Control Administrator's Guide11 Advanced Configuration p.267

7. Restart Control:pbs-control-manager restart

8. Login to the machine where the Simulator component is installed.

9. Edit the file PC_HOME/pbs-control-simulator/etc/pbssim/environment.

10. Change the value of PBS_SIM_ANALYTICS_HOST to the new hostname of the Analyze component.

11. Save the file.

12. Edit the file PC_HOME/pbs-control-simulator/etc/pbssim/pbssim.conf.

13. Change the value of analytics_host to the new hostname of the Analyze component.

14. Save the file.

15. Restart Control:pbs-control-manager restart

11.4.3 Reset Analyze and Simulation Data

Delete All Simulation and Model DataRemove simulation and model data before resetting the Analyze dataset.

The following steps must be done as root or as a user with sudo permissions using the sudo command.

When the Analyze dataset is discarded, all simulation and model data must be deleted as well.

1. Login to the machine where the Simulator component has been installed.

2. Execute the following command to remove all simulation and model data from Control:pbs-sim-agent --operation delete_all --auth-token placeholder

Simulation and model data is removed when the following message is displayed:PBS Sim data reset complete.

Reset the Analyze Accounting Log DataDiscard Analyze HPC complex data and reset the Data Collectors.

The following commands must be executed as root or as a user with sudo privileges using the sudocommand.

Discard all HPC complex accounting log and scheduler log data that has been forwarded by the DataCollectors. Optionally, all node information forwarded by the Data Collectors can also be discarded. Afterthe reset is complete, Data Collectors will again forward all HPC complex accounting and scheduler logs.Node information (pbsnodes -av) is also forwarded if the user chooses to discard it. Lastly, the Analyzedatabase password is reset back to the default of “pbsworks@123”.

1. Login to the machine where Analyze is installed.

2. Stop Control by issuing the command:pbs-control-manager stop

3. Execute the following command:

Proprietary Information of Altair Engineering

Page 273: Control Administrator's Guide - Altair PBS Works · Altair Control 2019.1 Administrator's Guide iv • Altair Embed SE™ ©1989-2019 (formerly solidThinking Embed® SE) • Altair

Control Administrator's Guide11 Advanced Configuration p.268

pbsa-data-reset

The message Delete all nodedetails configuration files? (Y/N) is displayed.

4. Choose one of the following options:

• Enter Y to remove all node information from Analyze and completely refresh the dataforwarded by the Data Collector (HPC complex accounting logs, scheduler logs, and pbsnodes-av information).

• Enter N to retain all node information and only refresh the HPC complex accounting andscheduler logs forwarded by the Data Collector.

5. Use the pbsa-data-password command to change the default database password:pbsa-data-password pbsworks@123 NEW_PASSWORD

Where NEW_PASSWORD is a secure site-specific password.

See AlsoAnalyze Data Forwarded by the Data CollectorReset the Analyze Dataset

Proprietary Information of Altair Engineering

Page 274: Control Administrator's Guide - Altair PBS Works · Altair Control 2019.1 Administrator's Guide iv • Altair Embed SE™ ©1989-2019 (formerly solidThinking Embed® SE) • Altair

Control Administrator's Guide11 Advanced Configuration p.269

11.5 Advanced Configuration for Cloud

11.5.1 Access Cloud AnalyticsView graphs of Docker container monitoring through a Grafana interface.

1. Click the Cloud tab.

2. Click Monitoring.The Cloud Monitoring interface is displayed.

11.5.2 Access the CloudFlow InterfaceAccess the CloudFlow interface to follow the application workflows.

1. Click the Cloud tab.

2. Click Workflows.A CloudFlow user interface is displayed.

11.5.3 Change the Hostname of the Cloud ComponentChange the hostname of the Cloud component when it is moved to a new host.

The following steps must be done as root or as a user with sudo permissions using the sudo command.

1. Login to the machine where the Control Web Interface is installed.

2. Edit the file PC_HOME/config/api_gateway/nginx/nginx.conf.

3. Change the Cloud component hostname by updating the existing value of server for upstreamPBSCloud in the format of hostname:port. upstream PBSCloud{ server dm-01:9980; }

4. Save the file.

5. Edit the file PC_HOME/config/api_gateway/nginx/conf.d/model.conf.

6. Change the value of $CLOUD_IO_URL to the new hostname in the format hostname:port.set $CLOUD_IO_URL "http://dm-01:9980/control/";

7. Save the file.

8. Edit the file PC_HOME/config/shared/token_handler_config/token_validator/RequestValidatorConfig.json.

9. Change the value of pbscloud_url to the new hostname in the format hostname:port."pbscloud_url":"http://dm-01:9980/control/"

10. Save the file.

11. Restart Control:

Proprietary Information of Altair Engineering

Page 275: Control Administrator's Guide - Altair PBS Works · Altair Control 2019.1 Administrator's Guide iv • Altair Embed SE™ ©1989-2019 (formerly solidThinking Embed® SE) • Altair

Control Administrator's Guide11 Advanced Configuration p.270

pbs-control-manager restart

Proprietary Information of Altair Engineering

Page 276: Control Administrator's Guide - Altair PBS Works · Altair Control 2019.1 Administrator's Guide iv • Altair Embed SE™ ©1989-2019 (formerly solidThinking Embed® SE) • Altair

Troubleshooting 12

12 Troubleshooting

This chapter covers the following:

• 12.1 Logging (p. 272)

• 12.2 Trace an Issue (p. 294)

• 12.3 Troubleshoot Adding a Cluster (p. 296)

• 12.4 Troubleshoot Port Conflicts (p. 298)

• 12.5 Troubleshoot a Web Page Rendering Issue in IE11 (p. 299)

• 12.6 Troubleshoot an UnknownHostKey Error (p. 300)

• 12.7 Troubleshoot Cloud Bursting DNS Errors (p. 301)

• 12.8 Troubleshoot an Out of Memory Parser Issue (p. 302)

• 12.9 Troubleshoot Simulation Failures on SLES 12 SP2 (p. 303)

• 12.10 Troubleshoot Simulation Failures on SUSE/SLES (p. 304)

Page 277: Control Administrator's Guide - Altair PBS Works · Altair Control 2019.1 Administrator's Guide iv • Altair Embed SE™ ©1989-2019 (formerly solidThinking Embed® SE) • Altair

Control Administrator's Guide12 Troubleshooting p.272

12.1 LoggingGeneral logging information.

Logging behavior for the following components and services can be modified:

• Web Interface

◦ Authentication

◦ Gateway

◦ Licensing

◦ Notifications

• Monitor

• Analyze

• Simulator and Configure

Default Logging LevelThe default logging level is set to informational providing messages that highlight the progress of theservice at a coarse-grained level.

12.1.1 Location of Log FilesUse log files to troubleshoot problems with Control.

Installer Log FilesObtain detailed logging information for the installation process by navigating to PC_INSTALLER.

A log file is written to this directory whenever Control is installed, therefore each installation log fileis timestamped. During the installation process, information is written to this file as each serviceis installed. Service installation is done asynchronously, so messages pertaining to all services areinterspersed throughout this log file. Installation messages are also written to a log file that is specificto each service. These log files are located at:

PC_HOME/logs/SERVICE>/SERVICE_node_config.log

Where SERVICE is the name of the service.

Installer Report FileEvery time Control is installed or uninstalled a file called pbswinstall_op_report is appended withparameter information used during the installation/uninstallation process. The file is created inthe directory /var/spool. The contents of this file may be valuable for diagnosing issues with theinstallation/uninstallation of Control.

Shared Resources Log FilesLog files for shared resources (Java libraries, Apache Web Server, Tomcat) can be found at: PC_HOME/logs/shared

Proprietary Information of Altair Engineering

Page 278: Control Administrator's Guide - Altair PBS Works · Altair Control 2019.1 Administrator's Guide iv • Altair Embed SE™ ©1989-2019 (formerly solidThinking Embed® SE) • Altair

Control Administrator's Guide12 Troubleshooting p.273

Component and Service Log File Location

Component Service Log File Location

Web Interface PC_HOME/logs/pcgui/tomcat

Message Router PC_HOME/logs/pbsworks-messagerouter

Gateway PC_HOME/logs/api_gateway

Licensing PC_HOME/logs/license

Notification PC_HOME/logs/wsbridge

Web Interface

Authentication PC_HOME/logs/ams

Analyze Server PC_HOME/logs/pbsa/logs

Data Collector PC_HOME/logs/dc/logs

Simulator PC_HOME/logs/pbsworks-admin

Monitor PC_HOME/logs/monitor/services

Configure PC_HOME/logs/pbsworks-admin

Cloud http://hostname:5601 where hostname is thehostname of the machine where the Cloud componentis installed.

12.1.2 Web Interface Logging

Change the Authentication Service Logging LevelChange the logging level to troubleshoot authentication.

The following steps must be done as root or as a user with sudo permissions using the sudo command.

1. Login to the machine where the Web Interface is installed.

2. Edit the file PC_HOME/config/ams/logging-config/log4j.properties.

3. Find the following line in the file:log4j.logger.com.altair=info, R

4. Change the value of log4j.logger.com.altair from "info" to the appropriate logging level.

• OFF - designated the highest possible ranking and is intended to turn off logging.

Proprietary Information of Altair Engineering

Page 279: Control Administrator's Guide - Altair PBS Works · Altair Control 2019.1 Administrator's Guide iv • Altair Embed SE™ ©1989-2019 (formerly solidThinking Embed® SE) • Altair

Control Administrator's Guide12 Troubleshooting p.274

• FATAL - designates very severe error events that will presumably lead the application toabort.

• ERROR - designates error events that might still allow the application to continue running.

• WARN - designates potentially harmful situations.

• INFO - designates informational messages that highlight the progress of the application atcoarse-grained level.

• DEBUG - designates fine-grained informational events that are most useful to debug anapplication.

• TRACE - designates finer-grained informational events than the DEBUG.

• ALL - designates the lowest possible rank and is intended to turn on all logging.

5. Restart Control:pbs-control-manager restart

Change the Gateway Service Logging LevelChange the logging level to troubleshoot the Gateway.

The following steps must be done as root or as a user with sudo permissions using the sudo command.

1. Login to the machine where the Web Interface is installed.

2. Edit the file PC_HOME/config/api_gateway/logging-config/log4j.properties.

3. Find the following line in the file:log4j.logger.com.altair=info, R

4. Change the value of log4j.logger.com.altair from "info" to the appropriate logging level.

• OFF - designated the highest possible ranking and is intended to turn off logging.

• FATAL - designates very severe error events that will presumably lead the application toabort.

• ERROR - designates error events that might still allow the application to continue running.

• WARN - designates potentially harmful situations.

• INFO - designates informational messages that highlight the progress of the application atcoarse-grained level.

• DEBUG - designates fine-grained informational events that are most useful to debug anapplication.

• TRACE - designates finer-grained informational events than the DEBUG.

• ALL - designates the lowest possible rank and is intended to turn on all logging.

5. Restart Control:pbs-control-manager restart

Proprietary Information of Altair Engineering

Page 280: Control Administrator's Guide - Altair PBS Works · Altair Control 2019.1 Administrator's Guide iv • Altair Embed SE™ ©1989-2019 (formerly solidThinking Embed® SE) • Altair

Control Administrator's Guide12 Troubleshooting p.275

Change the Licensing Service Logging LevelChange the logging level to troubleshoot licensing.

The following steps must be done as root or as a user with sudo permissions using the sudo command.

1. Login to the machine where the Web Interface is installed.

2. Edit the file PC_HOME/config/license/log4j.properties.

3. Find the following line in the file:log4j.logger.com.altair.pbsworks.license=INFO, licenseActivityLogAppender

4. Change the value of log4j.logger.com.altair.pbsworks.license from "INFO" to theappropriate logging level.

• OFF - designated the highest possible ranking and is intended to turn off logging.

• FATAL - designates very severe error events that will presumably lead the application toabort.

• ERROR - designates error events that might still allow the application to continue running.

• WARN - designates potentially harmful situations.

• INFO - designates informational messages that highlight the progress of the application atcoarse-grained level.

• DEBUG - designates fine-grained informational events that are most useful to debug anapplication.

• TRACE - designates finer-grained informational events than the DEBUG.

• ALL - designates the lowest possible rank and is intended to turn on all logging.

5. Find the following line in the file:log4j.logger.com.altair.pbsworks.license.api=INFO, licenseApiActivityLogAppender

6. Change the value of log4j.logger.com.altair.pbsworks.license.api from "INFO" to theappropriate logging level.

• OFF - designated the highest possible ranking and is intended to turn off logging.

• FATAL - designates very severe error events that will presumably lead the application toabort.

• ERROR - designates error events that might still allow the application to continue running.

• WARN - designates potentially harmful situations.

• INFO - designates informational messages that highlight the progress of the application atcoarse-grained level.

• DEBUG - designates fine-grained informational events that are most useful to debug anapplication.

• TRACE - designates finer-grained informational events than the DEBUG.

• ALL - designates the lowest possible rank and is intended to turn on all logging.

7. Restart Control:pbs-control-manager restart

Proprietary Information of Altair Engineering

Page 281: Control Administrator's Guide - Altair PBS Works · Altair Control 2019.1 Administrator's Guide iv • Altair Embed SE™ ©1989-2019 (formerly solidThinking Embed® SE) • Altair

Control Administrator's Guide12 Troubleshooting p.276

Change the Notification Service Logging LevelChange the logging level to troubleshoot the Notification Service.

The following steps must be done as root or as a user with sudo permissions using the sudo command.

1. Login to the machine where the Web Interface is installed.

2. Edit the file PC_HOME/config/wsbridge/log4js.json.

3. Find the following lines in the file:"categories": { "default": { "appenders": [ "wsbridge" ], "level": "info" } }

4. Change the value of level from "info" to the appropriate logging level.

• OFF - designated the highest possible ranking and is intended to turn off logging.

• FATAL - designates very severe error events that will presumably lead the application toabort.

• ERROR - designates error events that might still allow the application to continue running.

• WARN - designates potentially harmful situations.

• INFO - designates informational messages that highlight the progress of the application atcoarse-grained level.

• DEBUG - designates fine-grained informational events that are most useful to debug anapplication.

• TRACE - designates finer-grained informational events than the DEBUG.

• ALL - designates the lowest possible rank and is intended to turn on all logging.

5. Restart Control:pbs-control-manager restart

12.1.3 Configure and Simulator Logging

Change the Configure and Simulator Logging LevelChange the logging level to troubleshoot the Configure and Simulator components.

The following steps must be done as root or as a user with sudo permissions using the sudo command.

1. Login to the machine where the Configure and/or Simulate Simulator is installed.

2. Edit the file PC_HOME/pbsworks-admin/etc/environment.

3. Add the following line to the file:PBSWADMIN_LOG_LEVEL=

4. Set the value of PBSWADMIN_LOG_LEVEL to the appropriate logging level.

• DEBUG - fine-grained informational events that are most useful to debug an application.

• INFO - informational messages highlighting the progress of the application at a coarse grainedlevel.

• WARNING - potentially harmful situations such as failures and exceptions.

Proprietary Information of Altair Engineering

Page 282: Control Administrator's Guide - Altair PBS Works · Altair Control 2019.1 Administrator's Guide iv • Altair Embed SE™ ©1989-2019 (formerly solidThinking Embed® SE) • Altair

Control Administrator's Guide12 Troubleshooting p.277

5. Restart Control:pbs-control-manager restart

12.1.4 Monitor Logging

Monitor LoggingMonitor logging levels are set for specific Java packages. By default, the logging level is set to “info” forall packages and provides a nominal amount of logging.

If you want increase the details of logging for a package, then you need to change the logging level forthe corresponding package.

Below is a list of the Monitor Java packages and the types of classes the package contains.

Package Type of classes

com.altair.monitor.nextgen Base package of Monitor

com.altair.monitor.nextgen.cache Cache refreshers classes

com.altair.monitor.nextgen.command Action, Jobs and Nodes commands

com.altair.monitor.nextgen.core

com.altair.monitor.nextgen.data Data providers base class

com.altair.monitor.nextgen.data.cache App to EHCache cache managers classes

com.altair.monitor.nextgen.data.services Job and node summary parsers classes

com.altair.monitor.nextgen.model

com.altair.monitor.nextgen.notifications Notification and refresh all classes

com.altair.monitor.nextgen.queue Action response update queues classes

com.altair.monitor.nextgen.services Service classes for endpoints

com.altair.monitor.nextgen.session Session handling classes

com.altair.monitor.nextgen.utils General utility clases

com.altair.monitor.nextgen.ws Service endpoint classes

Proprietary Information of Altair Engineering

Page 283: Control Administrator's Guide - Altair PBS Works · Altair Control 2019.1 Administrator's Guide iv • Altair Embed SE™ ©1989-2019 (formerly solidThinking Embed® SE) • Altair

Control Administrator's Guide12 Troubleshooting p.278

Change the Monitor Logging LevelChange the logging level to troubleshoot the Monitor component.

The following steps must be done as root or as a user with sudo permissions using the sudo command.

1. Login to the machine where Monitor is installed.

2. Edit the file PC_HOME/config/monitor/services/loggerConfig.xml.

3. Scroll down until you find the Java package for which the logging level is going to be changed.In this example, the logging level for the base package of Monitor is being changed.<category name="com.altair.monitor.nextgen"> <priority value="info" /> <!-- <priority value="debug" /> --> <!-- <priority value="trace" /> --></category>

4. Change the logging level for the Java package to:

Log Level Action

debug 1. Comment out <<priority value="info" />> by prefixing the line with <!–and suffixing the line with --> .

2. Remove the <!-- and --> surrounding <priority value="debug">

trace 1. Comment out <<priority value="info" />> by prefixing the line with <!–and suffixing the line with --> .

2. Remove the <!-- and --> surrounding <priority value="trace">

Tip: The logging level can be changed to any valid log4j logging level in addition to"debug" or "trace".

• OFF - designated the highest possible ranking and is intended to turn off logging.

• FATAL - designates very severe error events that will presumably lead the application toabort.

• ERROR - designates error events that might still allow the application to continue running.

• WARN - designates potentially harmful situations.

• INFO - designates informational messages that highlight the progress of the application atcoarse-grained level.

• DEBUG - designates fine-grained informational events that are most useful to debug anapplication.

• TRACE - designates finer-grained informational events than the DEBUG.

• ALL - designates the lowest possible rank and is intended to turn on all logging.

This example changes the logging level from INFO to DEBUG.<category name="com.altair.monitor.nextgen"> <!-- <priority value="info" /> -->

Proprietary Information of Altair Engineering

Page 284: Control Administrator's Guide - Altair PBS Works · Altair Control 2019.1 Administrator's Guide iv • Altair Embed SE™ ©1989-2019 (formerly solidThinking Embed® SE) • Altair

Control Administrator's Guide12 Troubleshooting p.279

<priority value="debug" /> <!-- <priority value="trace" /> --></category>

5. Restart Control:pbs-control-manager restart

12.1.5 Analyze Logging

Change the Analyze Log File SizeDefine the Analyze maximum log file size.

The following steps must be done as root or as a user with sudo permissions using the sudo command.

By default, Analyze generates a new log file once the size of the current log file exceeds 100MB.

1. Login to the machine where the Analyze component is installed.

2. Edit the file PC_HOME/config/pbsa/config/log4j.properties.

3. Locate the following line in the file:log4j.appender.frameworkRollingFile.MaxFileSize=100MB

4. Set the value of log4j.appender.frameworkRollingFile.MaxFileSize to the appropriate log filesize in megabytes.

5. Restart Control.pbs-control-manager restart

Change the Analyze Logging LevelChange the logging level to troubleshoot Analyze.

The following steps must be done as root or as a user with sudo permissions using the sudo command.

1. Login to the machine where the Analyze component is installed.

2. Edit the file PC_HOME/config/pbsa/config/log4j.properties.

3. Locate the following line in the file:log4j.rootLogger=INFO, frameworkRollingFile

4. Change the value of log4j.rootLogger from "INFO" to the appropriate logging level:

• OFF - designated the highest possible ranking and is intended to turn off logging.

• FATAL - designates very severe error events that will presumably lead the application toabort.

• ERROR - designates error events that might still allow the application to continue running.

• WARN - designates potentially harmful situations.

• INFO - designates informational messages that highlight the progress of the application atcoarse-grained level.

• DEBUG - designates fine-grained informational events that are most useful to debug anapplication.

Proprietary Information of Altair Engineering

Page 285: Control Administrator's Guide - Altair PBS Works · Altair Control 2019.1 Administrator's Guide iv • Altair Embed SE™ ©1989-2019 (formerly solidThinking Embed® SE) • Altair

Control Administrator's Guide12 Troubleshooting p.280

• TRACE - designates finer-grained informational events than the DEBUG.

• ALL - designates the lowest possible rank and is intended to turn on all logging.

5. Restart Control.pbs-control-manager restart

Change the Maximum Number of Analyze Log Backup FilesDefine the maximum number of Analyze log backup files to keep.

The following steps must be done as root or as a user with sudo permissions using the sudo command.

By default, Analyze keeps a maximum of 1000 files as backup.

1. Login to the machine where the Analyze component is installed.

2. Edit the file PC_HOME/config/pbsa/config/log4j.properties.

3. Locate the following line in the file:log4j.appender.frameworkRollingFile.MaxBackupIndex=1000

4. Set log4j.appender.frameworkRollingFile.MaxBackupIndex to the appropriate number ofbackup files to save.

5. Restart Control.pbs-control-manager restart

Analyze Error Log MessagesAnalyze log message indicating a need for action.

Message Explanation

Problem in invoking parser Unable to start the Parser. Contact systemssupport.

Exception while creating the recoverydirectory

Unable to create a Recovery directory. Contactsystems support.

Exception while creating the recovery logfile

Unable to create a Recovery file Contact systemssupport.

Unable to parse the date format Internal error. Contact systems support.

error while retrieving key performancelogs

PBSA logs performance information such as howlong it takes to parse, cache, and the overall timefor both activities. PBSA was unable to retrievethis information. No action is necessary.

Caching Status URL cannot be empty Contact systems support.

error while retrieving disk space Disk space issues. Contact systems support.

Proprietary Information of Altair Engineering

Page 286: Control Administrator's Guide - Altair PBS Works · Altair Control 2019.1 Administrator's Guide iv • Altair Embed SE™ ©1989-2019 (formerly solidThinking Embed® SE) • Altair

Control Administrator's Guide12 Troubleshooting p.281

Message Explanation

Eh-Cache configuration for parsercachedoesn't exists

Internal error. Contact systems support.

unable to initialise Terracotta Eh cache. Internal error. Contact systems support.

Unable read table metadata whilegenerating csv files

Internal error. Contact systems support.

File couldn't be moved Internal error. Contact systems support.

unable to write output file Internal error. Contact systems support.

A wrong value is configured a To Date There is an issue with the format of a To Datein a node detail configuration file. PBSA will logadditional information to help determine whichnode detail entry is invalid. Edit the node detailsfile and correct the issue. The corrected entry willbe picked up in the next Parser cycle.

Error while generating checksum forcustomresources.xml file.

Internal error. Contact systems support.

Error while gernerating md5 checksum. Internal error. Contact systems support.

Error while writting checksum file. Internal error. Contact systems support.

Error while validation checksum forcustomresources.xml file.

Internal error. Contact systems support.

Custom resource file updated, please do adata reset everytime customresources.xmlfile is updated.

A custom resource was added to the customresource configuration file, but a data resetwas not performed. Reset the PBSA data setand restart PBSA. Once PBSA is restarted, allaccounting logs will be reparsed.

error in loading custom hook class PBSA could not load the Java class associated witha custom resource. Contact systems support.

Skipping configuration of hook forRESOURCE

There was a problem with configuring a PBASAhook for the custom resource denoted byRESOURCE. Contact systems support.

Error while configuring custom resource There was a problem with configuring a customresource hook. Contact systems support.

Unable read table metadata for rc_event_resource_occupancy_base table

Internal error. Contact systems support.

Proprietary Information of Altair Engineering

Page 287: Control Administrator's Guide - Altair PBS Works · Altair Control 2019.1 Administrator's Guide iv • Altair Embed SE™ ©1989-2019 (formerly solidThinking Embed® SE) • Altair

Control Administrator's Guide12 Troubleshooting p.282

Message Explanation

Error while calculating proportinatewalltime

Internal error. Contact systems support.

Error while converting values Internal error. Contact systems support.

Error while parsing for unknown reasons Internal error. Contact systems support.

Error while parsing record, please seestack trace

An exception occurred while parsing a PBSaccounting log record. Contact systems support.

Found multiple accounting events loggedin a single accounting record, hencediscarding it

Duplicate records have been found in the PBSaccounting log for a re-run job. The first record isprocessing and the second record is ignored. Noaction is required.

Job array main job information is notsupported, writing the following record tounsupported folder.

The main container accounting log record for ajob array is ignored. PBSA will only process thesubjobs of the job array. No action is required.

Could not determine user or groupinformation reliably, hence discarding it.

The user or group under which the job ranis missing, therefore PBSA will discard theaccounting log record. No action is required.

Could not find job start time informationin accounting record, hence discarding it.

The start time of the job is missing, thereforePBSA will discard the accounting log record. Noaction is required.

Could not find job qtime information inaccounting record, hence discarding it.

The time the job was queued is missing, thereforePBSA will discard the accounting log record. Noaction is required.

Could not find job end time information inaccounting record, hence discarding it.

The time the job ended is missing, therefore PBSAwill discard the accounting log record. No action isrequired.

Could not find job used memory informationin accounting record, hence discarding it.

The memory used by the job is missing, thereforePBSA will discard the accounting log record. Noaction is required.

Could not find job execution hostsinformation in accounting record, hencediscarding it.

Execution host information is missing, thereforePBSA will discard the accounting log record. Noaction is required.

Could not connect to the database Internal error. Contact systems support.

Error while checking configuration fileexistence

Either the exit status or custom groupconfiguration files are missing. Contact systemssupport.

Proprietary Information of Altair Engineering

Page 288: Control Administrator's Guide - Altair PBS Works · Altair Control 2019.1 Administrator's Guide iv • Altair Embed SE™ ©1989-2019 (formerly solidThinking Embed® SE) • Altair

Control Administrator's Guide12 Troubleshooting p.283

Message Explanation

Incorrect Date Format Encountered: Internal error. Contact systems support.

Monetdb database backup failed. Internal error. Contact systems support.

Exception while converting the memoryattributes to KB

Internal error. Contact systems support.

error occurred while populatingconfigurations

There is an issue with either the exit status orcustom group configuration file. Review anychanges you may have made to these files andmake modifications as necessary.

Please set the VM parameter for theconfiguration file

Internal error. Contact systems support.

Error while reading ThreadPool ScaleFactor from parser configuration.

Internal error. Contact systems support.

Caching Status URL cannot be empty. Internal error. Contact systems support.

Unable to send request for caching status. Verify that the PBSA Service is running. If it is,then contact systems support.

Complete parse cycle URL cannot be empty. Internal error. Contact systems support.

Unable to send request for complete parsecycle.

Internal error. Contact systems support.

Start Parse Cycle URL cannot be empty Internal error. Contact systems support.

Unable to send request for start parsecycle.

Internal error. Contact systems support.

Error writing load history file Internal error. Contact systems support.

Please check the accounting log file name,it should be of the format 'yyyymmdd'

A file was found while parsing the PBS Professionalaccounting logs that does not adhere to thenaming convention YYYYMMDD. PBSA will ignorethis file and continue parsing. No action required.

Future date accounting log file A file was found while parsing the PBS Professionalaccounting logs that adheres to the namingconvention YYYYMMDD, however the date is afuture date. PBSA will ignore this file and continueparsing. No action required.

Error while creating start record Internal error. Contact systems support.

Proprietary Information of Altair Engineering

Page 289: Control Administrator's Guide - Altair PBS Works · Altair Control 2019.1 Administrator's Guide iv • Altair Embed SE™ ©1989-2019 (formerly solidThinking Embed® SE) • Altair

Control Administrator's Guide12 Troubleshooting p.284

Message Explanation

Error while fetch start value for S jobs Internal error. Contact systems support.

Error occured while executing report Internal error. Contact systems support.

Not able to load cache. Internal error. Contact systems support.

Eh-Cache configuration for query cachedoesn't exists.

Internal error. Contact systems support.

Error in holiday file in line NN There is a problem with the holiday configurationfile in the line denoted by NN. Edit the holiday fileand correct the issue. The corrected entry will bepicked up in the next Parser cycle.

Error while polulating Holiday cache. Internal error. Contact systems support.

Error while executing Post Parse cycleactivities

Internal error. Contact systems support.

Unable to complete Cache Priming. Internal error. Contact systems support.

OTB Cache Priming failed. Internal error. Contact systems support.

Error Occured for User: USERNAME Internal error. Contact systems support.

Problem occured while caching Internal error. Contact systems support.

Error reading Chart: Internal error. Contact systems support.

Exception while creating the directory Internal error. Contact systems support.

Not able to load pbsaconfig.properties. Internal error. Contact systems support.

Unable to retrieve licencs dates. Internal error. Contact systems support.

Exception Thrown ,could not connect to thedatabase

Internal error. Contact systems support.

Error while executing : QUERY Internal error. Contact systems support.

ERROR: NumberFormatException while parsingthe PROCESS_MONITROING_INTERVAL :

Internal error. Contact systems support.

Proprietary Information of Altair Engineering

Page 290: Control Administrator's Guide - Altair PBS Works · Altair Control 2019.1 Administrator's Guide iv • Altair Embed SE™ ©1989-2019 (formerly solidThinking Embed® SE) • Altair

Control Administrator's Guide12 Troubleshooting p.285

Analyze Informational Log MessagesAnalyze log messages highlighting the progress of the application at a coarse-grained level.

Message Explanation

**Starting PBS Analytics Parsercomponent****

The service responsible for running the Parser isstarting.

Analytics engine initialization is inprogress, hence skipping this parse cycle.

Parsing and caching of the accounting logs did notcomplete before another Parser cycle initiated (itran for more than 24 hours). The newest Parsercycle is skipped and those accounting logs thatare not processed are picked up by the next day'sParser cycle.

PBS accounting log parse cycle startedDATETIME

Parsing of the PBS Professional accounting logsis starting on the date and time denoted byDATETIME.

Total Time Taken to parse DC accountinglogs : SS

The time SS, in seconds, it took to parse theaccounting logs transported from the DataCollector denoted by DC.

PBS accounting log parsing completed forDC

Parsing of the accounting logs transported fromthe Data Collector denoted by DC is completed.

Number of complete parse cycle : NN The number of parsing cycles denoted by NN thathave completed.

Parser has completed parsing theaccounting logs.

Parsing of the accounting logs is complete.

Analytics engine is now being initializedwith the parsed accounting log data.Please do not login to the Web Applicationuntil this process is complete.

Data must be loaded into the PBSA database andcached before the Web Application is available.

To check for completion of theinitialization process, view the WebApplication log file pbsworks.log.

A message will be written to the pbsworks.log filewhen the Web Application is available.

Parser configuration location : PATH The directory denoted by PATH is where the PBSAconfiguration files (node details, holiday, exitstatus) are stored.

Configuration file for parser FILE Location of the parser configuration file denotedby FILE. This file contains information about whenthe daily Parser cycle is scheduled to run.

Proprietary Information of Altair Engineering

Page 291: Control Administrator's Guide - Altair PBS Works · Altair Control 2019.1 Administrator's Guide iv • Altair Embed SE™ ©1989-2019 (formerly solidThinking Embed® SE) • Altair

Control Administrator's Guide12 Troubleshooting p.286

Message Explanation

Server shut down called when parsing is inprogress

Parsing is in progress when the PBSA service isstopped or restarted. Parsing will continue whenthe PBSA service is started.

Number of records in recovery : NN The number of accounting log entries, denoted byNN, that had issues and were written to a file inthe Recovery directory.

Record will be written into recovery An issue was found with an accounting log entry.The issue is logged and the entry is written to afile in the Recovery directory. A Recovery directoryis created for each Data Collector. The Recoverydirectories are located in PC_HOME/portal/services/pbsa/parser/modules/pbsdata. Asystems administrator can edit these files andcorrect the issue with the accounting log entry.The corrected entries are parsed and loaded intothe PBSA database during the next Parser cycle.

The parser is running in RECOVERY MODE The Parser is processing the accounting logrecords that were written to the Recoverydirectory. Files are written to the Recoverydirectory when an issue is found with anaccounting log entry. A Recovery directory iscreated for each Data Collector. The Recoverydirectories are located in PC_HOME/portal/services/pbsa/parser/modules/pbsdata. Asystems administrator can edit these files andcorrect the issue with the accounting log entry.The corrected entries are parsed and loaded intothe PBSA database during the next Parser cycle.

Going to delete corrected recovery fileFILE

Accounting log entries in the Recovery directoryare corrected, parsed, and loaded into the PBSAdatabase, therefore the Recovery file denoted byFILE is being deleted.

The parser is running in NORMAL MODE The Parser is running in NORMAL MODE meaningthat newly transported accounting logs are beingparsed.

The parser is running in RUNNING MODE PBSA writes information about running jobs toa directory during the parsing process. The filesstored in the Running Job directory are processedduring RUNNING MODE.

Proprietary Information of Altair Engineering

Page 292: Control Administrator's Guide - Altair PBS Works · Altair Control 2019.1 Administrator's Guide iv • Altair Embed SE™ ©1989-2019 (formerly solidThinking Embed® SE) • Altair

Control Administrator's Guide12 Troubleshooting p.287

Message Explanation

Rerun Jobs encountered multiple times : Duplicate records have been found in the PBSaccounting log for a re-run job. The first record isprocessing and the second record is ignored. Noaction is required.

No files to parse in : PATH No accounting logs were written to either theRecovery directory or to the Running Jobsdirectory.

Parsing FILE log file The accounting log file denoted by FILE is beingparsed.

File is parsed: FILE The accounting log file denoted by FILE is parsed.

Deletion failed for FILE Failed to delete an accounting log file that wasparsed. No action is necessary. This file will beignored in the next Parser cycle.

Number of running jobs : NN This message is displayed once during a parsingcycle and shows how many running jobs, denotedby NN, were found during the parsing cycle.

Deleting Zombie Jobs PBSA is deleting Zombie Jobs - jobs that aremissing an end record (“E”), and are missingresource information on the start record (“S”).PBSA ignores these jobs and continues parsing. Noaction required.

Zombie Jobs deletion complete PBSA is finished deleting Zombie Job - jobs thatare missing an end record (“E”), and are missingresource information on the start record (“S”).

Parser Cache initialised successfully Caching is complete.

Importing Data for finished jobs Loading the PBSA database with finished jobinformation.

Importing Data for node data Loading the PBSA database with node detailinformation.

Data import finished for node data Loading of the PBSA database with node detailinformation is complete.

Importing Data Loading of data into the PBSA database isbeginning.

Proprietary Information of Altair Engineering

Page 293: Control Administrator's Guide - Altair PBS Works · Altair Control 2019.1 Administrator's Guide iv • Altair Embed SE™ ©1989-2019 (formerly solidThinking Embed® SE) • Altair

Control Administrator's Guide12 Troubleshooting p.288

Message Explanation

Importing Data for running jobs Loading the PBSA database with running jobinformation.

Importing Data for running jobs completed Loading of the PBSA database with running jobinformation is complete.

Importing Data for node data for runningjobs

Loading the PBSA database with node detailinformation for running jobs.

No data available for Import No data is available to be loaded into the PBSAdatabase.

Import of Data Complete Loading of PBSA database is complete.

Reading config file : FILE Reading the PBSA configuration file denoted byFILE.

Finished Reading file :FILE Finished reading the PBSA configuration filedenoted by FILE.

*****No custom resource configured ***** There are no custom resources configured for thissite.

Going to read custom resourceconfiguration

Beginning to process custom resources (PBSAHooks).

You have provided a custom hook classfor this resource, going to verify classname : CLASS

Verifying that the class denoted by CLASS that isassociated with the custom resource exists.

Successfully able to load hook class Custom resource class is found and loaded.

Custom Resource processing finished Processing of custom resources is complete.

Time taken to execute Query is MS This message indicates that a query was executed(a chart was displayed via the Web Application)and it took MS milliseconds to execute the query.

Request received for starting parsingcycle.

The Parser is sending a message to the PBSAWeb Application indicating that parsing isstarting. Users will be blocked from using the WebApplication while parsing is in process.

Request received for completing parsingcycle.

The Parser is sending a message to the PBSA WebApplication indicating that parsing is completedand caching is beginning. Users will be blockedfrom using the Web Application while caching is inprocess.

Proprietary Information of Altair Engineering

Page 294: Control Administrator's Guide - Altair PBS Works · Altair Control 2019.1 Administrator's Guide iv • Altair Embed SE™ ©1989-2019 (formerly solidThinking Embed® SE) • Altair

Control Administrator's Guide12 Troubleshooting p.289

Message Explanation

Query Cache initialised successfully. PBSA is caching the queries associated with thePBSA default charts and user charts.

Unable to find config directory forholiday configurations.

The directory where the PBSA configuration filesare located is missing. Contact systems support.

Reading node details file to populateavailabilty information

Reading the node detail configuration files.

OTB Cube caching started PBSA default chart are being cached.

OTB Cube caching completed in Caching of PBSA default charts is completed.

Initialization of the analytics enginecompleted successfully in MM minutes. Youmay now login to the PBSA web applicationusing the URL http://<hostname>:<port>/pbsworks

Parsing and caching is complete. The WebApplication is available.

Executing: CHART Caching the chart denoted by CHART

Total number of licensing records : NN Total number of licensing records denoted by NNfound during the parser cycle.

Analyze Warning Log MessagesAnalyze log messages indicating a potential issue and action may be required.

Message Explanation

System memory check failed, Exit status:EXITCODE

PBSA was unable to check for the availability ofmemory for running PBSA. No action is necessary.

Problem while importing : DATA There was an issue while loading the PBSAdatabase. Contact systems support.

ATTR and DATATYPE are mandatory elementswhile configuring new custom resource,hence skipping this resource

Mandatory XML elements for defining a customresource are missing. This custom resource willnot be processed. Add the missing XML elementsto the custom resource file, discard any parseddata, and reparse all PBS Professional accountinglogs.

Record will be written into recovery An issue was found with an accounting log entry.The issue is logged and the entry is written to a

Proprietary Information of Altair Engineering

Page 295: Control Administrator's Guide - Altair PBS Works · Altair Control 2019.1 Administrator's Guide iv • Altair Embed SE™ ©1989-2019 (formerly solidThinking Embed® SE) • Altair

Control Administrator's Guide12 Troubleshooting p.290

Message Explanation

file in the Recovery directory. A Recovery directoryis created for each Data Collector. The Recoverydirectories are located in PC_HOME/portal/services/pbsa/parser/modules/pbsdata. Editthe Recovery file and correct the issue with theaccounting log entry. The corrected entries areparsed and loaded into the PBSA database duringthe next Parser cycle during RECOVERY MODE.

Record will be written into unsupportedfolder

PBSA found an accounting log entry that containsinvalid data. The issue is logged and the entryis written to a file in the unsupported directory.An unsupported directory is created for eachData Collector. The unsupported directories arelocated in INSTALL_DIR/portal/services/pbsa/parser/modules/pbsdata. View the contents ofa file in the unsupported directory to analyze theissue. You may need to make changes to the PBSProfessional configuration to make the missing/invalid PBS attribute available in the accountinglog record.

No value for attribute PBSA found an accounting log entry that containsinvalid data. The issue is logged and the entry iswritten to a file in the unsupported directory.

config file not present A PBSA configuration file is missing. No action isnecessary.

Not providing these node detailsconfigurations will result in productshowing you limited set of reports. Pleaseconsult Administrator Guide for details ofcharts and features provided / affected bythis configuration.

No action required. For more information on howthe absence of a node detail information affectsthe PBSA default charts.

Server Shutdown called, going out of parsedirectory.

Parsing was in progress when the PBSA servicewas stopped or restarted. The accounting log filethat is currently being parsed will be completed.Parsing will continue when the PBSA service isstarted. No action required.

Empty parse cycle for dc : DC No accounting logs were transported for the DataCollector denoted by DC. No action required.

Error while fetching timezone difference PBSA was unable to determine the timezone ofthe Data Collector. The accounting logs will be

Proprietary Information of Altair Engineering

Page 296: Control Administrator's Guide - Altair PBS Works · Altair Control 2019.1 Administrator's Guide iv • Altair Embed SE™ ©1989-2019 (formerly solidThinking Embed® SE) • Altair

Control Administrator's Guide12 Troubleshooting p.291

Message Explanation

stamped with the timezone of the PBSA server. Noaction required.

Unable to createrc_event_resource_occupancy table.

Internal error. Contact systems support.

Unable to create nodes table. Internal error. Contact systems support.

12.1.6 Cloud Logging

Access the Cloud LogsMonitor Cloud by viewing its logs through a Kibana interface.

1. Click the Cloud tab.

2. Click Logs.A Kibana user interface is displayed which can be used to view and monitor Cloud logs.

Change the Docker Logging ConfigurationSet the maximum size of the Docker log file and the maximum log files to be kept.

Configure your Docker logging system to avoid consuming a lot of disk space and to rotate Docker logs.For more information see https://docs.docker.com/config/containers/logging/configure/.

Parameters that can be set to configure Docker logging:

container_log_max_sizeThe maximum size of the log before it is rolled. A positive integer plus a modifier representing theunit of measure (k, m, or g). Defaults to -1 (unlimited).

container_log_max_fileThe maximum number of log files that can be present. If rolling the logs creates excess files,the oldest file is removed. Only effective when max-size is also set. Must be a positive integer.Defaults to 1.

By default, 10 log files are retained and the maximum size of a log file is set to 10MB.

1. Create an environment variable for the Cloud version:export CLOUD_VERSION=2019.1

2. Export the current installation location of the Cloud kard.

• If you installed Cloud using the Control installer and installed in the default location:export KARD_PATH=\/var/spool/pbsworks/$CLOUD_VERSION/pbscontrol/home/cloudio/kard

Proprietary Information of Altair Engineering

Page 297: Control Administrator's Guide - Altair PBS Works · Altair Control 2019.1 Administrator's Guide iv • Altair Embed SE™ ©1989-2019 (formerly solidThinking Embed® SE) • Altair

Control Administrator's Guide12 Troubleshooting p.292

• If you installed Cloud using the Control installer and installed in a non-default location:export KARD_PATH=DIR/cloudio/kard

Replace DIR with the path to the Control home directory.

• If you installed Cloud manually:export KARD_PATH=DIR/kard

Replace DIR with the path where Cloud is installed.

3. Navigate to the directory $KARD_PATH/current.

4. Create/edit the file meta.yml and add the following lines to the file:

5. Change the maximum number of log files to be kept:container_log_max_file: MAX_LOGS_KEEP

6. Change the maximum size of a log file:container_log_max_size: MAX_LOG_SIZE

7. Restart Control:pbs-control-manager restart

Change the Elasticsearch Log Retention PeriodSet the maximum number of days that Elasticsearch logs should be retained.

Elasticsearch is a search and analytics engine used to store all logs for Cloud. Since all Cloud logs arestored in Elasticsearch, the size of its logs may grow rapidly. To avoid consuming a lot of disk space,configure the number of days that Elasticsearch logs should be retained.

Parameters that can be set to configure Elasticsearch logging:

elasticsearch_retention_daysThe maximum number of days that Elasticsearch logs should be retained.

By default, Elasticsearch logs are retained for 30 days.

1. Create an environment variable for the Cloud version:export CLOUD_VERSION=2019.1

2. Export the current installation location of the Cloud kard.

• If you installed Cloud using the Control installer and installed in the default location:export KARD_PATH=\/var/spool/pbsworks/$CLOUD_VERSION/pbscontrol/home/cloudio/kard

• If you installed Cloud using the Control installer and installed in a non-default location:export KARD_PATH=DIR/cloudio/kard

Replace DIR with the path to the Control home directory.

• If you installed Cloud manually:export KARD_PATH=DIR/kard

Replace DIR with the path where Cloud is installed.

3. Navigate to the directory $KARD_PATH/current.

4. Create/edit the file meta.yml and add the following lines to the file:

Proprietary Information of Altair Engineering

Page 298: Control Administrator's Guide - Altair PBS Works · Altair Control 2019.1 Administrator's Guide iv • Altair Embed SE™ ©1989-2019 (formerly solidThinking Embed® SE) • Altair

Control Administrator's Guide12 Troubleshooting p.293

5. Change the number of days that Elasticsearch logs should be retained.elasticsearch_retention_days: MAX_RENTION_DAYS

6. Restart Control:pbs-control-manager restart

Proprietary Information of Altair Engineering

Page 299: Control Administrator's Guide - Altair PBS Works · Altair Control 2019.1 Administrator's Guide iv • Altair Embed SE™ ©1989-2019 (formerly solidThinking Embed® SE) • Altair

Control Administrator's Guide12 Troubleshooting p.294

12.2 Trace an Issue

12.2.1 Diagnosis ScriptA script that captures Control log and system data to be used for diagnostics.

Namepbswinstall_agent.py

Syntaxpbswinstall_agent.py diagnosis

DescriptionCapture log data for all Control components and services, as well as system data, running processesand installer log files. This information is useful when troubleshooting issues with Control. By default thelast 24 hours worth of log files are collected.

This command must be executed as root or as a user with sudo privileges using the sudo command onthe machine where Control is installed.

Parameter FileA parameter file PC_INSTALLER/pbswinstall.conf can be updated to configure the number of days oflog files to collect. The file contains the two parameters:[diagnosis]journalctl_start = todayjournalctl_end = tomorrow

To collect logs for additional days, these parameters can be set to a date in the format YYYY-MM-DD.

Table 18: Parameter Examples and the Number of Days of Log Files Collected

journalctl_start journalctl_end Days Log Files are Collected

today tomorrow Last 24 hours of log files are collected.

today today Only today's logs files are collected.

2018-11-15 2018-11-05 Log files from Nov 5th to Nov 15th are collected.

Example

PC_INSTALLER/pbswinstall_agent.py diagnosis

Proprietary Information of Altair Engineering

Page 300: Control Administrator's Guide - Altair PBS Works · Altair Control 2019.1 Administrator's Guide iv • Altair Embed SE™ ©1989-2019 (formerly solidThinking Embed® SE) • Altair

Control Administrator's Guide12 Troubleshooting p.295

12.2.2 Use the Diagnosis ScriptUse the Diagnsosis script to gather log files and system data to help troubleshoot issues with Control.

1. Navigate to the PC_INSTALLER directory.

2. Enter the following command:source source_install_env && ./framework_init

3. Run the script:./pbswinstall_agent.py diagnosis

A tarball file is created in the current working directory calledHOSTNAME_diag_report_DATE_TIME.tar.gz where DATE and TIME is the file creation timestampin the format YYYY-MM-DD-HH-MM-SS.MMMMM.

Proprietary Information of Altair Engineering

Page 301: Control Administrator's Guide - Altair PBS Works · Altair Control 2019.1 Administrator's Guide iv • Altair Embed SE™ ©1989-2019 (formerly solidThinking Embed® SE) • Altair

Control Administrator's Guide12 Troubleshooting p.296

12.3 Troubleshoot Adding a ClusterWhile attempting to add a cluster, a cluster already exists message is displayed.

ConditionAfter attempting to add a cluster via the Monitor tab, the message Cluster name already exists orServer name/IP name already exists is displayed.

CauseCluster name already exists indicates a cluster has already been added with the same ClusterName.

RemedyRetry adding the cluster using a unique Cluster Name.

1. Click Manage Clusters.A dialog box appears where information about the cluster is entered.

Figure 88: Manage Clusters

2. For Cluster Name, enter a unique name for the cluster.

The cluster name must be 20 characters or less.

3. Continue filling in the rest of the information necessary to add the cluster.

CauseServer name/IP name already exists indicates a cluster has already been added with the same Hostname, but a different Cluster Name.

RemedyFind the previously added cluster based on its hostname.

1. Click Manage Clusters.HPC clusters that have been previously added are listed on the left hand side of the dialog box.

Proprietary Information of Altair Engineering

Page 302: Control Administrator's Guide - Altair PBS Works · Altair Control 2019.1 Administrator's Guide iv • Altair Embed SE™ ©1989-2019 (formerly solidThinking Embed® SE) • Altair

Control Administrator's Guide12 Troubleshooting p.297

Figure 89: Manager Clusters

2. Select a cluster by clicking on the name of the cluster.The form to the right of the list of clusters is populated with cluster information.

3. Examine the value of Host for the cluster.

4. Repeat the previous two steps until you find the cluster with a hostname that matches thehostname of the cluster you were trying to add.

This is the cluster that you want to monitor.

CauseIf neither of the above remedies resolve the issue and the message Cluster name already existsor Server name/IP name already exists continues to display after attempting to add the cluster, aManager may have revoked your access to the cluster.

RemedyContact your Control administrator.

Proprietary Information of Altair Engineering

Page 303: Control Administrator's Guide - Altair PBS Works · Altair Control 2019.1 Administrator's Guide iv • Altair Embed SE™ ©1989-2019 (formerly solidThinking Embed® SE) • Altair

Control Administrator's Guide12 Troubleshooting p.298

12.4 Troubleshoot Port ConflictsResolve port conflicts encountered while installing Control.

Default ports must be free to successfully install Control. When a port conflict is encountered duringinstallation, a message is logged and also displayed to the user who is installing the product. Theerror message that is displayed is Cannot start <servicename> service, port <port number> iscurrently in use.

Currently, there is no way to change the port used by all of the Control services, so the process usingthe port must be stopped.

1. Determine which process is using the port using an appropriate Linux command such as netstator lsof.

2. Stop the process that is using the port.

3. Uninstall Control.

4. Re-install Control.

See AlsoControl Components, Services and PortsUninstall Control

Proprietary Information of Altair Engineering

Page 304: Control Administrator's Guide - Altair PBS Works · Altair Control 2019.1 Administrator's Guide iv • Altair Embed SE™ ©1989-2019 (formerly solidThinking Embed® SE) • Altair

Control Administrator's Guide12 Troubleshooting p.299

12.5 Troubleshoot a Web Page Rendering Issue inIE11After logging into Control, the web page is rendering incorrectly.

Any users that login to Control using IE11 must disable the Compatibility View mode so that IE11renders the Control web pages correctly.

1. Launch Internet Explorer 11.

2. Click located in the upper right-hand corner of the web page.

3. Click Compatibility View settings.

4. Disable Display intranet sites in Compatibility View.

5. Click Close.

Proprietary Information of Altair Engineering

Page 305: Control Administrator's Guide - Altair PBS Works · Altair Control 2019.1 Administrator's Guide iv • Altair Embed SE™ ©1989-2019 (formerly solidThinking Embed® SE) • Altair

Control Administrator's Guide12 Troubleshooting p.300

12.6 Troubleshoot an UnknownHostKey ErrorWhile adding a cluster, an UnknownHostKey error is thrown.

Control assumes that SSH key-based authentication, between the machine where the Control servicesare installed and the machine hosting the PBS Server, has been configured for the user account that isthe PBSWorks Service User (as defined during the installation of Control). Control attempts to connectto the machine hosting the PBS Server using the Service User username and SSH key. Errors similar tothe following indicate that there is a problem with the SSH key:UnknownHostKey: <hostname>. RSA key fingerprint is 26:79:cd:30:09:de:c1:54:58:fc:2b:1f:a4:b9:75:fa

A possible cause may be a mismatch between the type of the SSH key defined for the PBSWorksService User and the one that is contained in the known_hosts file on the machine hosting Control.

1. Login to the machine hosting Control.

2. Enter the following command to determine the type of SSH key that has been generated:ssh-keygen -l -f <SSH_PUBLIC_KEY_PATH>

Where <SSH_PUBLIC_KEY_PATH> is the path to the public SSH key of the PBSWorks ServiceUser.

You will see output similar to the following indicating the type of SSH key that has beengenerated.2048 5d:08:01:09:88:c4:10:a1:4d:21:35:6a:b1:fd:50:44 <SSH_PUBLIC_KEY_PATH>(RSA)

3. Check the type of key stored in the known_hosts file.

4. If the key types are different, enter the following command:ssh-keyscan -t <TYPE> <HOSTNAME> >> known_hosts

where <TYPE> is the type of SSH key that has been generated (output from Step 2) and<HOSTNAME> is the hostname of the PBS Server.

Proprietary Information of Altair Engineering

Page 306: Control Administrator's Guide - Altair PBS Works · Altair Control 2019.1 Administrator's Guide iv • Altair Embed SE™ ©1989-2019 (formerly solidThinking Embed® SE) • Altair

Control Administrator's Guide12 Troubleshooting p.301

12.7 Troubleshoot Cloud Bursting DNS ErrorsTry to force Docker to use your site's DNS server when DNS errors are encountered during cloudbursting.

DNS errors encountered while cloud bursting are displayed in the Cloud Notification center (accessed byclicking located in the upper right-hand of the Cloud tab). You will see errors like:

Failed to establish a new connection: [Errno -2] Name or service not known

Try the following to resolve the DNS issue:

1. Login to the machine hosting Cloud.

2. Create the file /etc/docker/daemon.json.

3. Edit the file.

4. Add the following lines to the file:{ "dns": ["YOUR_NAMESERVER1", "YOUR_NAMESERVER2"]}

5. Save the file.

6. Restart the Docker daemon:systemctl restart docker

Proprietary Information of Altair Engineering

Page 307: Control Administrator's Guide - Altair PBS Works · Altair Control 2019.1 Administrator's Guide iv • Altair Embed SE™ ©1989-2019 (formerly solidThinking Embed® SE) • Altair

Control Administrator's Guide12 Troubleshooting p.302

12.8 Troubleshoot an Out of Memory Parser IssueConditionWhile parsing a large accounting and scheduler log dataset, the following exception occurs:java.lang.OutOfMemoryError: GC overhead limit exceeded

CauseThe default parsing memory is insufficient to parse large accounting and scheduler log datasets.

Remedy

Increase the parsing memory and restart Control.

1. Edit the PC_EXEC/pbsa/scripts/pbsworks-parser-start.sh file.

2. Search for the following line in the file:JAVA_OPTS="-Xms256m -Xmx1024m"

3. Increase the parser memory by changing JAVA_OPTS to:JAVA_OPTS="-Xms256m -Xmx12g"

4. Restart Control.pbs-control-manager restart

Proprietary Information of Altair Engineering

Page 308: Control Administrator's Guide - Altair PBS Works · Altair Control 2019.1 Administrator's Guide iv • Altair Embed SE™ ©1989-2019 (formerly solidThinking Embed® SE) • Altair

Control Administrator's Guide12 Troubleshooting p.303

12.9 Troubleshoot Simulation Failures on SLES 12SP2ConditionSimulations fail and messages similar to the following error are logged:Invalid state detected for simulation: SIM_FAILED. Redirecting to the simulation list page.

CausePAS 13.2 packages a Python which was compiled without certain SSL libraries.

RemedyA workaround is to recompile the Python installed with PAS.

1. Verify that an up-to-date version of PyOpenSSL (17x+) is installed on the machine hostingControl:rpm -qa pyOpenSSL*

Note: You can find the release history of PyOpenSSL at https://pypi.org/project/pyOpenSSL/#history

2. Navigate to /opt/altair/pas/13.2/pas/python.

3. sudo to root:sudo su

4. Recompile python:make clean && ./configure --prefix=$PWD && make && make install

5. Restart PAS:/etc/init.d/pas restart

Proprietary Information of Altair Engineering

Page 309: Control Administrator's Guide - Altair PBS Works · Altair Control 2019.1 Administrator's Guide iv • Altair Embed SE™ ©1989-2019 (formerly solidThinking Embed® SE) • Altair

Control Administrator's Guide12 Troubleshooting p.304

12.10 Troubleshoot Simulation Failures on SUSE/SLESConditionSimulations fail and messages similar to the following error are logged:xxx simulation failed stage 2, error retrieving results from PAS, no file xxx.results exists

CauseOn SLES/SUSE platforms, intermittently PAS 13.2 fails to deploy properly, such that the zip and unziputilities are not placed in the correct location.

Remedy

A workaround is to recompile the PAS included Python and replace the zip and unzip utilities with linksto the system installed version of those utilities:

Note: The workaround assumes that zip and unzip are installed in /usr/bin/.

1. Navigate to the PAS installed python directory:cd /opt/altair/pas/13.2/pas/python

2. Rebuild PAS python:make clean && ./configure --prefix=/opt/altair/pas/13.2/pas/python && make && make install

3. Save the PAS zip utility:mv /opt/altair/pas/13.2/pas/bin/Linux-x86_64/zip \/opt/altair/pas/13.2/pas/bin/Linux-x86_64/PAS_packaged_zip

4. Link to the system installed zip:ln -s /usr/bin/zip /opt/altair/pas/13.2/pas/bin/Linux-x86_64/zip

5. Save the PAS unzip utility:mv /opt/altair/pas/13.2/pas/bin/Linux-x86_64/unzip \/opt/altair/pas/13.2/pas/bin/Linux-x86_64/PAS_packaged_unzip

6. Link to the system installed unzip:ln -s /usr/bin/unzip /opt/altair/pas/13.2/pas/bin/Linux-x86_64/unzip

Proprietary Information of Altair Engineering

Page 310: Control Administrator's Guide - Altair PBS Works · Altair Control 2019.1 Administrator's Guide iv • Altair Embed SE™ ©1989-2019 (formerly solidThinking Embed® SE) • Altair

Install Control WithoutInternet Connectivity A-1

A-1 Install Control Without Internet Connectivity

Install Control on a machine that does not have access to the internet. This requires a manualinstallation of Cloud.

This chapter covers the following:

• A-1.1 Install socat (p. 306)

• A-1.2 Install Docker (p. 307)

• A-1.3 Install MongoDB (p. 308)

• A-1.4 Run the Cloud Prerequisite Script (p. 309)

• A-1.5 Install Control (p. 310)

• A-1.6 Install Cloud Manually (p. 312)

• A-1.7 Install the Analyze Data Collector (p. 320)

• A-1.8 Complete Upgrade and Post-Installation Steps (p. 322)

Page 311: Control Administrator's Guide - Altair PBS Works · Altair Control 2019.1 Administrator's Guide iv • Altair Embed SE™ ©1989-2019 (formerly solidThinking Embed® SE) • Altair

Control Administrator's GuideA-1 Install Control Without Internet Connectivity p.306

A-1.1 Install socatInstall the socat package a third-party software dependency.

Use the following instructions for downloading and installing socat:

• For RHEL/CentOS:sudo yum install socat

• For SLES/SUSE, see https://software.opensuse.org/download.html?project=network%3Autilities&package=socat

Proprietary Information of Altair Engineering

Page 312: Control Administrator's Guide - Altair PBS Works · Altair Control 2019.1 Administrator's Guide iv • Altair Embed SE™ ©1989-2019 (formerly solidThinking Embed® SE) • Altair

Control Administrator's GuideA-1 Install Control Without Internet Connectivity p.307

A-1.2 Install DockerInstall Docker from a package or binaries.

• Installation must be done as root or as a user with sudo permissions using the sudo command.

Note: If you want to run Docker as a non-root user, please see instructions to add thisuser to the Docker group. For more information see Manage Docker as a non-root user.

1. Login to a machine with internet connectivity.

2. Download the Docker package or binaries using these instructions:

• For RHEL/CentOS, use these instructions: Install Docker CE from binaries.

• For SLES/SUSE, use these instructions: Install Docker EE from a package.

3. Login to the machine where Control is going to be installed.

4. Copy the Docker package or binaries from the machine with internet connectivity using acommand such as scp.

5. Choose one of the following options to install and start Docker:

• For RHEL/CentOS, use these instructions: Install Docker CE from binaries.

• For SLES/SUSE, use these instructions: Install Docker EE from a package.

6. Verify that the file /usr/bin/docker-init exists.

7. If the file does not exist:

a) Login to a machine with internet connectivity.

b) Download the tarball for the latest version of Docker from the Docker repository:

https://download.docker.com/linux/static/stable/x86_64/

c) Login to the machine where Control is going to be installed.

d) Copy the tarball file from the machine with internet connectivity using a command such asscp.

e) Extract the archive to /tmp:tar zxvf docker-18.06.1-ce.tgz -C /tmp

f) Execute the command:cp /tmp/docker/docker-init /usr/bin/

Proprietary Information of Altair Engineering

Page 313: Control Administrator's Guide - Altair PBS Works · Altair Control 2019.1 Administrator's Guide iv • Altair Embed SE™ ©1989-2019 (formerly solidThinking Embed® SE) • Altair

Control Administrator's GuideA-1 Install Control Without Internet Connectivity p.308

A-1.3 Install MongoDBInstall MongoDB for use by the Web Interface, Analyze, Configure, and Simulator component for datastorage.

Attention: If the Monitoring component is being installed stand-alone, then MongoDB doesnot need to be installed. Otherwise, if you are installing any of the other components (WebInterface, Analyze, Configure, Simulator), MongoDB must be installed.

Installation must be done as root or as a user with sudo permissions using the sudo command.

Restriction: Only MongoDB 3.4 and 3.6 are supported.

Due to legal reasons, the installation of MongoDB could not be incorporated into the 2019.1 installer,therefore it must be manually installed, wherever the Web Interface, Analyze, Configure, and Simulatorcomponents are installed. Once MongoDB is installed, no other configuration is necessary.

1. Login to a machine with internet connectivity.

2. Download the MongoDB tarball from the MongoDB Download Center.

3. Login to the machine where Control is going to be installed.

4. Copy the MongoDB tarball from the machine with internet connectivity using a command such asscp.

5. Choose one of the following options:

• Follow these instructions to install MongoDB 3.4: Install MongoDB Community Edition FromTarball 3.4

• Follow these instructions to install MongoDB 3.6: Install MongoDB Community Edition FromTarball 3.6

Proprietary Information of Altair Engineering

Page 314: Control Administrator's Guide - Altair PBS Works · Altair Control 2019.1 Administrator's Guide iv • Altair Embed SE™ ©1989-2019 (formerly solidThinking Embed® SE) • Altair

Control Administrator's GuideA-1 Install Control Without Internet Connectivity p.309

A-1.4 Run the Cloud Prerequisite ScriptPrerequisite check for Cloud.

The following steps must be done as root or as a user with sudo permissions using the sudo command.

Download and run the Cloud prerequisite script to check the following Cloud prerequisites beforeinstallation:

• Presence of Docker and its version

• Space for the images (/var/lib/docker)

• Connectivity with the Cloud Docker repository

Note: The script will complain that it cannot connect to the Cloud Docker repository if youare installing Cloud manually because the machine where you are installing Control does nothave internet connectivity.

1. Download or obtain the pbscloud_preq.sh script using your usual Altair support channels.

2. Run the script to verify that all prerequisites for installing Cloud have been met:./pbscloud_preq.sh

Proprietary Information of Altair Engineering

Page 315: Control Administrator's Guide - Altair PBS Works · Altair Control 2019.1 Administrator's Guide iv • Altair Embed SE™ ©1989-2019 (formerly solidThinking Embed® SE) • Altair

Control Administrator's GuideA-1 Install Control Without Internet Connectivity p.310

A-1.5 Install ControlInstall all Control components except Cloud using the Advanced installation type.

Review the installation prerequisites before proceeding with the installation.

Note: Installer log files are available in PC_INSTALLER/logs.

Several components use MongoDB for data storage, therefore there will be multiple database instancesrunning once this installation is complete. Monitor and PBS Professional use Postgres for data storage,therefore there will be two Postgres database instances running after this installation is complete.

Every time Control is installed or uninstalled a file called pbswinstall_op_report is appended withparameter information used during the installation/uninstallation process. The file is created inthe directory /var/spool. The contents of this file may be valuable for diagnosing issues with theinstallation/uninstallation of Control.

1. Login to the machine where Control is to be installed.

2. Download or obtain the installer tarballAltairControl_Installer_2019.1_628_20190413_010457.tar.gz using your usual Altairsupport channels.

3. Navigate to the directory where the installer tarball was copied by changing your workingdirectory.

4. Extract the contents of the tarball:tar -xvf AltairControl_Installer_2019.1_628_20190413_010457.tar.gz

A directory called AltairControl_Installer is created.

5. Navigate to the AltairControl_Installer directory:cd AltairControl_Installer

6. Invoke the installer by running the installation binary:./run_installer

If a broken installation is detected then the following message is displayed:Detected an improperly uninstalled or existing/broken installationExisting Altair Control installation needs to be uninstalled. Would you like to proceed? ['yes' or 'no']

Enter yes and press ENTER. Once the broken installation is completely removed, the installationcontinues.

7. Read the introduction, verify the prerequisites for installation, and press ENTER.

8. Page through the license agreement by pressing ENTER until you are asked to accept its terms andconditions.

9. Accept the license agreement by entering 1 and pressing ENTER.

10. For Installation type, choose Advanced by entering 2 and pressing ENTER.

11. For component, choose to install all components except for Cloud by entering 1,2,3,4,5 andpressing ENTER.

Proprietary Information of Altair Engineering

Page 316: Control Administrator's Guide - Altair PBS Works · Altair Control 2019.1 Administrator's Guide iv • Altair Embed SE™ ©1989-2019 (formerly solidThinking Embed® SE) • Altair

Control Administrator's GuideA-1 Install Control Without Internet Connectivity p.311

12. For PBS Cloud Host, enter the hostname of the machine where Cloud will be installed and pressENTER.

13. For License Server Details, enter the LMX license server in the format port@hostname and pressENTER.

14. Choose to install both Analyze and the Data Collector by entering 3 and pressing ENTER.

15. For PBSWorks Service User, enter a username who will be the PBSWorks Service User and pressENTER.

The user must exist on the machine where you are installing Control. All Control services includingthe database instances run as this user.

16. For Analyse Data Collector FQDN, enter a unique name (FQDN) for the Data Collector.

17. For Exec Directory, enter where you would like to install the execution binaries (PC_EXEC) or pressENTER to use the default location.

18. For Home Directory, enter where you would like to install the configuration files (PC_HOME) orpress ENTER to use the default location.

19. Review the pre-installation summary and then press ENTER to install the product.Message similar to those below are displayed as the installation progresses.Installing...-------------

This may take a while...Unzipping files into home, exec, and installer folders....

Progress is: 0% Web Interface Beginning PBSWorks Shared InstallationProgress is: 12% Web Interface Completed updating PBS Works Shared config files.........All milestones reached!!

20. Once the installation is complete, set the PBSWorks Service User's PAS password by issuing thecommand:su - SERVICE_USER -c "/opt/altair/pas/13.2/pas/pas-commands/bin/pas-passwd"

Where SERVICE_USER is the username of the PBSWorks Service User.

21. Enter the PBSWorks Service User's SSH password.

Proprietary Information of Altair Engineering

Page 317: Control Administrator's Guide - Altair PBS Works · Altair Control 2019.1 Administrator's Guide iv • Altair Embed SE™ ©1989-2019 (formerly solidThinking Embed® SE) • Altair

Control Administrator's GuideA-1 Install Control Without Internet Connectivity p.312

A-1.6 Install Cloud ManuallyManually install Cloud when there is no access to the internet.

When the machine where Control is installed is not connected to the internet, a manual installation ofCloud is necessary. A manual installation of Cloud requires at least the following steps:

1. Login to a machine having internet connectivity.

2. Install Docker.

3. Download and save the Cloud images (Docker is required to download and save the Cloudimages).

4. Login to the machine hosting Control (the machine without internet connectivity).

5. Copy the Cloud images from the machine with internet connectivity.

6. Load the Cloud images.

7. Install the Cloud containers.

8. Link Cloud with the PBS Works License Server.

9. Start and initialize Cloud.

Install DockerInstall Docker which is used to host Cloud.

Installing Docker for RHEL and CentOSInstall Docker for RHEL and CentOS platforms.

• Installation must be done as root or as a user with sudo permissions using the sudo command.

Note: If you want to run Docker as a non-root user, please see instructions to add thisuser to the Docker group. For more information see Manage Docker as a non-root user.

• You must have access to a machine that has internet connectivity and it must have the sameoperating system as the one where Control is going to be installed.

• 20GB free space on /var/lib/docker.

• For more information about installing Docker CE see Get Docker CE for CentOS.

1. Login to a machine that has internet connectivity.

2. Install required packages.yum install -y yum-utils

3. Setup the docker repository:yum-config-manager --add-repo \https://download.docker.com/linux/centos/docker-ce.repo

4. Install the packets and its dependencies:yum install docker-ce

Proprietary Information of Altair Engineering

Page 318: Control Administrator's Guide - Altair PBS Works · Altair Control 2019.1 Administrator's Guide iv • Altair Embed SE™ ©1989-2019 (formerly solidThinking Embed® SE) • Altair

Control Administrator's GuideA-1 Install Control Without Internet Connectivity p.313

5. Start Docker.systemctl start dockersystemctl enable docker

Proprietary Information of Altair Engineering

Page 319: Control Administrator's Guide - Altair PBS Works · Altair Control 2019.1 Administrator's Guide iv • Altair Embed SE™ ©1989-2019 (formerly solidThinking Embed® SE) • Altair

Control Administrator's GuideA-1 Install Control Without Internet Connectivity p.314

Installing Docker for SLES and SUSEInstall Docker for SLES and SUSE platforms.

• Installation must be done as root or as a user with sudo permissions using the sudo command.

Note: If you want to run Docker as a non-root user, please see instructions to add thisuser to the Docker group. For more information see Manage Docker as a non-root user.

• You must have access to a machine that has internet connectivity and it must have the sameoperating system as the one where Control is going to be installed.

• 20GB free space on /var/lib/docker.

• Docker CE is not supported on SLES, therefore Docker EE must be installed.

1. Login to a machine that has internet connectivity.

2. Install and start the latest version of Docker EE by following these instructions Get Docker EE forSLES.

3. Verify that the file /usr/bin/docker-init exists.

4. If the file does not exist:

a) Download the tarball for the latest version of Docker from the Docker repository:

https://download.docker.com/linux/static/stable/x86_64/

b) Extract the archive to /tmp:tar zxvf docker-18.06.1-ce.tgz -C /tmp

c) Execute the command:cp /tmp/docker/docker-init /usr/bin/

Download and Save the Cloud ImagesUse a Cloud download script to download and save Cloud images.

The installer script must be run on a machine having internet connectivity. This machine must haveDocker installed.

Use PBScloud_images_installer_2019.1.sh to download and save the Cloud images.

1. Login to the machine with internet connectivity where Docker was installed.

2. Download or obtain the PBScloud_images_installer_2019.1.sh script using your usual Altairsupport channels.

3. Executed the following command:./PBScloud_images_installer_2019.1.sh save

The Cloud images are downloaded and saved to a directory called PBScloud_images.

Proprietary Information of Altair Engineering

Page 320: Control Administrator's Guide - Altair PBS Works · Altair Control 2019.1 Administrator's Guide iv • Altair Embed SE™ ©1989-2019 (formerly solidThinking Embed® SE) • Altair

Control Administrator's GuideA-1 Install Control Without Internet Connectivity p.315

Load the Cloud ImagesUse a Cloud download script to load the Cloud images.

1. Login to the machine where Control is installed.

2. Copy the Cloud installer script PBScloud_images_installer_2019.1.sh from the machine withinternet connectivity using a command such as scp.

3. Copy the PBScloud_images directory from the machine with internet connectivity using acommand such as scp.

Make sure to copy the PBScloud_images directory to the same location asPBScloud_images_installer_2019.1.sh (same level).

4. Executed the following command:./PBScloud_images_installer_2019.1.sh load

The Cloud images are loaded.

Install the Cloud ContainersUse pkr to install the Cloud containers.

• The following steps must be done as root or as a user with sudo permissions using the sudocommand.

pkr is a Docker wrapper, designed to setup, install and manage the Cloud containers.

1. Create an environment variable for the Cloud version:export CLOUD_VERSION=2019.1

2. Create an environment variable for the installation location of Cloud:export INSTALL_DIR=DIR

Replace DIR with the path where you want to install Cloud.

3. Run the Pocker container used to setup and install Cloud:docker run -ti --network host --rm -e CLOUD_VERSION=$CLOUD_VERSION \-v /run/docker.sock:/run/docker.sock \-v $INSTALL_DIR/kard:/pkr/kard pbscloudio.azurecr.io/pkr:$CLOUD_VERSION bash

4. Create your deployment environment definition:pkr kard create $CLOUD_VERSION -e pbscontrol \--extra tag=$CLOUD_VERSION

5. Exit the container:exit

The cloud containers are now installed in the directory $INSTALL_DIR/kard.

Proprietary Information of Altair Engineering

Page 321: Control Administrator's Guide - Altair PBS Works · Altair Control 2019.1 Administrator's Guide iv • Altair Embed SE™ ©1989-2019 (formerly solidThinking Embed® SE) • Altair

Control Administrator's GuideA-1 Install Control Without Internet Connectivity p.316

Link Cloud with the PBS Works License ServerConfigure Cloud so that it can access the PBS Works license server.

The following steps must be executed as root or as a user with sudo permissions using the sudocommand.

Licensing for cloud bursting is done through the Control licensing service. This service is installedwhen the Web Interface component is installed. If Control has not been installed yet, you will need thehostname or IP address of the machine where it is going to be installed to configure Cloud for licensing.

1. Navigate to the directory $INSTALL_DIR/kard/current.

2. Create/edit the file meta.yml and add the following lines to the file:license_server: host=http://PC_WEB_INTERFACE_HOSTNAME:9100 auth_url=http://CLOUD_IP_ADDRESS:9980/control/

Replace PC_WEB_INTERFACE_HOSTNAME with the hostname/IP address of the machine wherethe Control Web Interface is installed. Replace CLOUD_IP_ADDRESS with the hostname/IP addressof the machine where Cloud is installed.

3. Save the file.

4. Create an environment variable for the Cloud version:export CLOUD_VERSION=2019.1

5. Launch a pkr container:docker run -ti --network host --rm -e CLOUD_VERSION=$CLOUD_VERSION \-v /run/docker.sock:/run/docker.sock \-v $INSTALL_DIR/kard:/pkr/kard pbscloudio.azurecr.io/pkr:$CLOUD_VERSION bash

6. Update the kard:pkr kard update

7. Update the Docker context:pkr kard make

8. Exit the container:exit

Start and Initialize CloudStart and initialize Cloud using pkr.

The following steps must be executed as root or as a user with sudo permissions using the sudocommand.

1. Launch a pkr container:docker run -ti --network host --rm -e CLOUD_VERSION=$CLOUD_VERSION \-v /run/docker.sock:/run/docker.sock \-v $INSTALL_DIR/kard:/pkr/kard pbscloudio.azurecr.io/pkr:$CLOUD_VERSION bash

2. Generate your environment:pkr kard make

3. Start Cloud:pkr start

Proprietary Information of Altair Engineering

Page 322: Control Administrator's Guide - Altair PBS Works · Altair Control 2019.1 Administrator's Guide iv • Altair Embed SE™ ©1989-2019 (formerly solidThinking Embed® SE) • Altair

Control Administrator's GuideA-1 Install Control Without Internet Connectivity p.317

4. Initialize Cloud by entering the command:pkr ext init full --domain_name <domain_name> \--username <admin_username> --password '<admin_password>' \--user_email '<admin_email>'

The values between the <> are used during the Control installation process to link Cloud andControl. You will need to replace these four values:

The domain_name does not have to be a true DNS domain, but it a group name for your project.It cannot include a period ( . ).

The admin_username and admin_password identify the credentials of the Cloud Administrator. Theadmin_username should not contain an asterisk ( * ). The admin_password must be at least 6characters in length and must contain at least one number.

Note: Alerts are sent to the Cloud Administrator's email denoted by <admin_email>when a cloud bursting resource alert limit is reached.

Below is an example:

pkr ext init full --domain_name chrysler --username tsmith --password 'password@123' --user_email '[email protected]'

5. Exit the container:exit

6. Navigate to the directory: PC_EXEC/api_gateway/scripts.

7. Encrypt the password (admin_password) entered while initializing the Cloud component (step 3above):./password_encryption.sh encrypt <admin_password>

The output of the script is the encrypted password.

8. Copy the encrypted password.

9. Navigate to the directory: PC_HOME/data/api_gateway.

10. Edit the file: Cloud_io_Credentials.json.

This file contains a key-value pair for "username" and "password" "username":"admin@altair", "password":"TM9RHX9nNLX0riiPM3/T6w=="

11. Replace the value of password with the encrypted password.

Note: Place the encrypted password between the double quotes.

12. Change the value of username to those values entered while initializing Cloud (step 3 above) inthe format admin_username@domain_name. "username":"tsmith@chrysler",

13. Save the file Cloud_io_Credentials.json.

14. Create a file cloudio_registration.json.

15. Edit the file and add the following lines:{ "host": "CLOUD_HOSTNAME", "username": "USERNAME",

Proprietary Information of Altair Engineering

Page 323: Control Administrator's Guide - Altair PBS Works · Altair Control 2019.1 Administrator's Guide iv • Altair Embed SE™ ©1989-2019 (formerly solidThinking Embed® SE) • Altair

Control Administrator's GuideA-1 Install Control Without Internet Connectivity p.318

"installed_parameters": { "node_list":["cloudio"] }}

Where CLOUD_HOSTNAME is the hostname/IP address of the machine where Cloud is installedand USERNAME is the <admin_username> entered while initializing Cloud (step 3 above).

Ex. tsmith

16. Save the file.

17. Execute the command:curl -X POST -v -H "Content-Type: application/json" \-d @./cloudio_registration.json http://localhost:9850/pcconfig/deployment

18. Restart Control:pbs-control-manager restart

Create a Service to Start and Stop CloudCreate a systemd service to automatically start and stop Cloud.

1. Download or obtain the pbscloud.service file using your usual Altair support channels.

2. Copy pbscloud.service to the directory /etc/systemd/system.

3. Create an environment variable for the Cloud version:export CLOUD_VERSION=2019.1

4. Export the current installation location of the Cloud kard.export KARD_PATH=$INSTALL_DIR/kard

5. Create a file that sets up the environment variables for starting and stopping Cloud:cat > /etc/sysconfig/pbscloud <<- EOFPKR_IMAGE=pbscloudio.azurecr.io/pkrPBSCLOUD_VERSION=$CLOUD_VERSIONPKR_VOLUME_OPTS="-v /run/docker.sock:/run/docker.sock -v $KARD_PATH:/pkr/kard"PKR_DOCKER_OPTS="--rm"EOF

6. Enable the service:systemctl enable pbscloud.service

7. Launch a pkr container:docker run -ti --network host --rm -e CLOUD_VERSION=$CLOUD_VERSION \-v /run/docker.sock:/run/docker.sock \-v $INSTALL_DIR/kard:/pkr/kard pbscloudio.azurecr.io/pkr:$CLOUD_VERSION bash

8. Stop Cloud:pkr stop

9. Exit the container:exit

You can now use the following systemctl commands to start and stop Cloud:

systemctl start pbscloud.service and systemctl stop pbscloud.service

Proprietary Information of Altair Engineering

Page 324: Control Administrator's Guide - Altair PBS Works · Altair Control 2019.1 Administrator's Guide iv • Altair Embed SE™ ©1989-2019 (formerly solidThinking Embed® SE) • Altair

Control Administrator's GuideA-1 Install Control Without Internet Connectivity p.319

Switch to the Locally Installed WebhelpConfigure Control to point to the locally installed webhelp when there is no internet connectivity.

By default, the Control webhelp (User Guide) is served through an Altair documentation portal. Internetconnectivity is necessary to access the portal. You can configure Control to point to webhelp that isinstalled locally when there is no internet connectivity.

1. Login to the machine where the Web Interface is installed.

2. Edit the file PC_HOME/config/api_gateway/nginx/conf.d/pc.conf.

3. Uncomment the below line of the file by removing the "#".#return 302 https://$http_host/pc/docs/index.htm;

4. Comment out the below line of the file by adding a "#" to the beginning of the line:return 302 https://connect.altair.com:443/CP/SA/NA/help/Control/2019.1/index.htm;

The line should look like this to comment it out.#return 302 https://connect.altair.com:443/CP/SA/NA/help/Control/2019.1/index.htm;

5. Restart Control:pbs-control-manager restart

Proprietary Information of Altair Engineering

Page 325: Control Administrator's Guide - Altair PBS Works · Altair Control 2019.1 Administrator's Guide iv • Altair Embed SE™ ©1989-2019 (formerly solidThinking Embed® SE) • Altair

Control Administrator's GuideA-1 Install Control Without Internet Connectivity p.320

A-1.7 Install the Analyze Data CollectorInstall the Analyze Data Collector on each PBS Professional complex.

• Installation must be done as root or as a user with sudo permissions using the sudo command.

• Installation must be done on a machine running on a supported platform.

• The Data Collector must have access to the machine hosting Analyze.

If your site has clusters with the same fully qualified domain name (FQDN), then the Analyze data isaggregated for both clusters under a single FQDN. To allow Analyze data for clusters having the sameFQDN to be tracked separately, a unique name must be provided while installing each Data Collector.

Note: Installer log files are available in PC_INSTALLER/logs.

Every time Control is installed or uninstalled a file called pbswinstall_op_report is appended withparameter information used during the installation/uninstallation process. The file is created inthe directory /var/spool. The contents of this file may be valuable for diagnosing issues with theinstallation/uninstallation of Control.

1. Login to the PBS Professional Server.

2. Download or obtain the installer tarballAltairControl_Installer_2019.1_628_20190413_010457.tar.gz using your usual Altairsupport channels.

3. Navigate to the directory where the installer tarball was copied by changing your workingdirectory.

4. Extract the contents of the tarball:tar -xvf AltairControl_Installer_2019.1_628_20190413_010457.tar.gz

A directory called AltairControl_Installer is created.

5. Navigate to the AltairControl_Installer directory:cd AltairControl_Installer

6. Invoke the installer by running the installation binary:./run_installer

If a broken installation is detected then the following message is displayed:Detected an improperly uninstalled or existing/broken installationExisting Altair Control installation needs to be uninstalled. Would you like to proceed? ['yes' or 'no']

Enter yes and press ENTER. Once the broken installation is completely removed, the installationcontinues.

7. Read the introduction, verify the prerequisites for installation, and press ENTER.

8. Page through the license agreement by pressing ENTER until you are asked to accept its terms andconditions.

9. Accept the license agreement by entering 1 and pressing ENTER.

10. For Installation type, choose Advanced by entering 2 and pressing ENTER.

Proprietary Information of Altair Engineering

Page 326: Control Administrator's Guide - Altair PBS Works · Altair Control 2019.1 Administrator's Guide iv • Altair Embed SE™ ©1989-2019 (formerly solidThinking Embed® SE) • Altair

Control Administrator's GuideA-1 Install Control Without Internet Connectivity p.321

11. For component, choose Analyze by entering 5 and pressing ENTER.

12. Choose to install the Data Collector by entering 2 and pressing ENTER.

13. For Web Interface, enter the hostname of the machine where the Control Web Interface isinstalled and press ENTER.

14. For License Server Details, enter the LMX license server in the format port@hostname and pressENTER.

15. For PBSA Server Host, enter the hostname of the machine where Analyze was installed.

16. For Analyse Data Collector FQDN, enter a unique name (FQDN) for the Data Collector.

17. For PBSWorks Service User, enter a username who will be the PBSWorks Service User and pressENTER.

The user must exist on the machine where you are installing Control. All Control services includingthe database instances run as this user.

18. For Exec Directory, enter where you would like to install the execution binaries (PC_EXEC) or pressENTER to use the default location.

19. For Home Directory, enter where you would like to install the configuration files (PC_HOME) orpress ENTER to use the default location.

20. Review the pre-installation summary and then press ENTER to install the product.Message similar to those below are displayed as the installation progresses.Installing... -------------This may take a while...Unzipping files into home, exec, and installer folders.... Progress is: 0% Analyze Beginning PBSWorks Shared InstallationProgress is: 12% Analyze Completed updating PBS Works Shared config files.........All milestones reached!!

21. Update the system path to include the path to the Analyze Commands:export PATH=$PATH:PC_EXEC/dc/scripts/pbsa-commands/bin

Proprietary Information of Altair Engineering

Page 327: Control Administrator's Guide - Altair PBS Works · Altair Control 2019.1 Administrator's Guide iv • Altair Embed SE™ ©1989-2019 (formerly solidThinking Embed® SE) • Altair

Control Administrator's GuideA-1 Install Control Without Internet Connectivity p.322

A-1.8 Complete Upgrade and Post-InstallationStepsComplete any upgrade and post-installation steps.

See AlsoComplete the UpgradePost-Installation Configuration

Proprietary Information of Altair Engineering

Page 328: Control Administrator's Guide - Altair PBS Works · Altair Control 2019.1 Administrator's Guide iv • Altair Embed SE™ ©1989-2019 (formerly solidThinking Embed® SE) • Altair

Start and Stop Cloud After aManual Installation A-2

A-2 Start and Stop Cloud After a Manual Installation

This chapter covers the following:

• A-2.1 Start Cloud (p. 324)

• A-2.2 Stop Cloud (p. 325)

• A-2.3 Restart Cloud (p. 326)

• A-2.4 Determine the Status of the Cloud Service (p. 327)

Page 329: Control Administrator's Guide - Altair PBS Works · Altair Control 2019.1 Administrator's Guide iv • Altair Embed SE™ ©1989-2019 (formerly solidThinking Embed® SE) • Altair

Control Administrator's GuideA-2 Start and Stop Cloud After a Manual Installation p.324

A-2.1 Start CloudStart the Cloud service after a manual installation.

Starting Cloud must be done as root or as a user with sudo permissions using the sudo command.

When your server hosting the Cloud component reboots, containers are restarted automatically. If youneed to manually start Cloud containers, please follow the below instructions.

1. Login to the machine where Cloud is installed.

2. Enter the following command to start Cloud:

systemctl start pbscloud.service

Proprietary Information of Altair Engineering

Page 330: Control Administrator's Guide - Altair PBS Works · Altair Control 2019.1 Administrator's Guide iv • Altair Embed SE™ ©1989-2019 (formerly solidThinking Embed® SE) • Altair

Control Administrator's GuideA-2 Start and Stop Cloud After a Manual Installation p.325

A-2.2 Stop CloudStop the Cloud service after a manual installation.

Stopping Cloud must be done as root or as a user with sudo permissions using the sudo command.

1. Login to the machine where Cloud is installed.

2. Enter the following command to stop Cloud:

systemctl stop pbscloud.service

Proprietary Information of Altair Engineering

Page 331: Control Administrator's Guide - Altair PBS Works · Altair Control 2019.1 Administrator's Guide iv • Altair Embed SE™ ©1989-2019 (formerly solidThinking Embed® SE) • Altair

Control Administrator's GuideA-2 Start and Stop Cloud After a Manual Installation p.326

A-2.3 Restart CloudRetart the Cloud service after a manual installation.

Retarting Cloud must be done as root or as a user with sudo permissions using the sudo command.

When your server hosting the Cloud component reboots, containers are restarted automatically. If youneed to manually restart Cloud containers, please follow below instructions.

1. Login to the machine where Cloud is installed.

2. Enter the following command to restart Cloud:

systemctl restart pbscloud.service

Proprietary Information of Altair Engineering

Page 332: Control Administrator's Guide - Altair PBS Works · Altair Control 2019.1 Administrator's Guide iv • Altair Embed SE™ ©1989-2019 (formerly solidThinking Embed® SE) • Altair

Control Administrator's GuideA-2 Start and Stop Cloud After a Manual Installation p.327

A-2.4 Determine the Status of the Cloud ServiceDetermine whether the Cloud service is up or down.

1. Login to the machine where Cloud is installed.

2. Enter the following command to display the status of the Cloud service:systemctl status pbscloud.service

Proprietary Information of Altair Engineering

Page 333: Control Administrator's Guide - Altair PBS Works · Altair Control 2019.1 Administrator's Guide iv • Altair Embed SE™ ©1989-2019 (formerly solidThinking Embed® SE) • Altair

Uninstall Cloud after a ManualInstallation A-3

A-3 Uninstall Cloud after a Manual Installation

Remove an instance of Cloud when it has been manually installed.

• You must stop Cloud before uninstalling it.

• Uninstallation must be done as root or as a user with sudo permissions using the sudo command.

Only use the following procedure when Cloud was installed manually. If Cloud was installed using theControl installer, then use the Control installer to remove Cloud.

1. Login to the machine where Cloud is installed.

2. Create an environment variable for the Cloud version:export CLOUD_VERSION=2019.1

3. Create an environment variable for the installation location of Cloud:export INSTALL_DIR=DIR

Replace DIR with the path where Cloud is installed.

4. Connect to the pkr container:docker run -ti --network host --rm -e CLOUD_VERSION=$CLOUD_VERSION \-v /run/docker.sock:/run/docker.sock \-v $INSTALL_DIR/kard:/pkr/kard pbscloudio.azurecr.io/pkr:$CLOUD_VERSION bash

5. Clean Cloud services:pkr clean

6. Purge Cloud images:pkr image purge

7. Exit the container:exit

8. Remove pkr container:docker image rm pbscloudio.azurecr.io/pkr:$CLOUD_VERSION

9. Remove the Cloud kard:rm -rf $INSTALL_DIR/kard

Page 334: Control Administrator's Guide - Altair PBS Works · Altair Control 2019.1 Administrator's Guide iv • Altair Embed SE™ ©1989-2019 (formerly solidThinking Embed® SE) • Altair

IndexAAnalyze 11, 28, 39, 308authentication

multi-factorDUO 258enable 258

Service Principal NameKerberos 257register 257verify 257

Single Sign-Ondisable 257enable 255Kerberos 255NT LAN Manager 255NTLM 255Service Principal Name 255SPN 255

CCloud 11, 28, 37, 37, 38, 42, 44, 307, 310, 312, 312, 314, 320, 328CloudFlow

Cloud 269configure 42, 44, 310, 320conventions

PC_EXEC 8PC_HOME 8

Ddeployment 13, 28, 35Docker 37, 37, 38, 307, 312, 312, 314DUO 258

HHyperWorks 11, 28

Iinstallation

Analyze 28Cloud 28, 37, 37, 38, 307, 312, 312, 314configure 42, 44, 310, 320Control 28, 42, 44, 310, 320

329

Page 335: Control Administrator's Guide - Altair PBS Works · Altair Control 2019.1 Administrator's Guide iv • Altair Embed SE™ ©1989-2019 (formerly solidThinking Embed® SE) • Altair

HyperWorks 28installers 28MongoDB 28, 39, 308monitor 42, 44, 310, 320PBS Application Services 28PBS Professional 28simulator 42, 44, 310, 320simulator application definitions 42, 44, 310, 320simulator solver 42, 44, 310, 320single machine 28web interface 42, 44, 310, 320

KKerberos 255, 257

Llogin access

default 35logs

Cloud 269, 291

MMongoDB 28, 39, 42, 44, 308, 310, 320monitor 42, 44, 310, 320

NNT LAN Manager 255NTLM 255

PPAS , See PBS Application ServicesPBS Application Services 11, 28PBS Professional 11, 28PC_EXEC 8, 42, 44, 310, 320PC_HOME 8, 42, 44, 310, 320pkr 37, 37, 38, 307, 312, 312, 314

SService Principal Name 255, 257, 257Service User 35, 39, 42, 44, 308, 310, 320services

analyze 13Analyze 86Cloud 13

330

Page 336: Control Administrator's Guide - Altair PBS Works · Altair Control 2019.1 Administrator's Guide iv • Altair Embed SE™ ©1989-2019 (formerly solidThinking Embed® SE) • Altair

configure 13Control 86MongoDB 13monitor 13restart

Control 89simulator 13start

Cloud 318, 324, 326Control 87

statusControl 90, 327

stopCloud 318, 325Control 88

web interface 13simulator 39, 42, 44, 308, 310, 320simulator application definitions 42, 44, 310, 320simulator solver 42, 44, 310, 320SPN 255, 257, 257SSO 255, 257, 257supported

browsers 11operating systems 11products

Analyze 11Cloud 11HyperWorks 11PBS Application Services 11PBS Professional 11

Uuninstall

Cloud 328Control 83, 84

Wweb interface 42, 44, 310, 320

331