Signaling Delivery Controller Bare Metal System … · Signaling Delivery Controller Bare Metal System Upgrade ... F5 Signaling Delivery Controller Bare Metal System Upgrade Guide

Proprietary and Confidential Information of F5 Networks

Signaling Delivery Controller Bare Metal System Upgrade Guide 5.1

Catalog Number: RG-016-51-31 Ver. 12

Publication Date: December 2017

F5 Signaling Delivery Controller Bare Metal System Upgrade Guide

[I] Proprietary and Confidential Information of F5

Networks

Legal Information

Copyright

© 2005-2017 F5 Networks, Inc. All rights reserved.

F5 Networks, Inc. (F5) believes the information it furnishes to be accurate and reliable. However, F5 assumes no responsibility for the use of this information, nor any infringement of patents or other rights of third parties which may result from its use. No license is granted by implication or otherwise under any patent, copyright, or other intellectual property right of F5 except as specifically described by applicable user licenses. F5 reserves the right to change specifications at any time without notice.

Trademarks

AskF5, F5, F5 [DESIGN], F5 Networks, OpenBloX, OpenBloX (design), Rosetta Diameter Gateway, Signaling Delivery Controller, SDC, Traffix, and Traffix [DESIGN] are trademarks or service marks of F5 Networks, Inc., in the U.S. and other countries, and may not be used without F5’s express written consent.

All other product and company names herein may be trademarks of their respective owners.

Patents

This product may be protected by one or more patents indicated at: http://www.f5.com/about/guidelines-policies/patents

Confidential and Proprietary

The information contained in this document is confidential and proprietary to F5 Networks. The information in this document may be changed at any time without notice.

About F5 Networks

F5 Networks (NASDAQ: FFIV) makes the connected world run better. F5 helps organizations meet the demands and embrace the opportunities that come with the relentless growth of voice, data, and video traffic, mobile workers, and applications—in the data center, the network, and the cloud. The world’s largest businesses, service providers, government entities, and consumer brands rely on F5’s intelligent services framework to deliver and protect their applications and services while ensuring people stay connected. For more information, visit www.F5.com or contact us at [email protected].

http://www.f5.com/about/guidelines-policies/patents

http://www.f5.com/about/guidelines-policies/patents

http://www.f5.com/

mailto:[email protected]


[II] Proprietary and Confidential Information of F5

Networks

About this Document Document Name: F5 Signaling Delivery Controller Bare Metal System Upgrade Guide

Catalog Number: RG-016-51-31 Ver. 12

Publication Date: December 2017

Document Objectives

This document describes the supported upgrades for the SDC 5.1 release installed on bare

metal.

Note: In this document, "server" and "machine" are used interchangeably.

Document History

Revision Number Change Description Change Location

Ver. 2 – November 2016 Added description of ports used

by the SDC

Port Settings Used by the SDC

Ver. 3 – January 2017 Added rollback procedure to a

previous CF. Added procedure for

manually configuring ports when

in mix-mode. Added information

about EMS Web UI screens

during an upgrade. Added section

to restart EMS NMS server

Performing an Upgrade

Rollback to a Previous CF,

Configuring the SDC Site

Ports. Working with the SDC

during an Upgrade. Restart

the EMS NMS Server


Rollback to a Previous CF

Ver. 4 – February 2017 Added ports used by the SDC.

Added prerequisites for EMS

Upgrade

Port Settings Used by the SDC,

Verify flowManager XML File

Size, Validate TC Timer

Parameter

Ver. 5 – March 2017 Updated SS7 pre-and post -

procedure. Updated SDC site

Copy the SS7 License and

Configuration Files,


[III] Proprietary and Confidential Information of F5

Networks

Revision Number Change Description Change Location

rollback procedure. Updated

operating system upgrade

procedure.


Rollback to 4.4, Upgrading the

Operating System

Ver. 6 – April 2017 Updated SDC site rollback

procedure.

Rolling Back an SDC Site

Upgrade to Release 4.4

Ver. 7 – May 2017 Updated the New ISO image

procedure.

New ISO Image

Ver. 8 – June 2017 Updated the rolling upgrade

process

Rolling Upgrade to a New

Release Version

Ver. 9 – August 2017 Added a note about SNMPv3

functionality while in mix-mode.

Updated the port descriptions

Modify Existing SNMP

Configurations, Modify

Existing SNMP

Configurations,


Ver. 10 – September 2017 Updated the port descriptions.

Added prerequisite (setting

external port range) for rolling

upgrade

Port Settings Used by the SDC,

Setting the External Port

Range

Ver. 11 – November 2017 Added note about copyFromIso

procedure. Added prerequisite for

rolling upgrade.

Uploading the New ISO image

for the Component Upgrade,

Updating the Salt Files

Ver. 12 – December 2017 Edited saltupgrade procedure for

rolling upgrade

Upgrading the Master

Installer

Conventions

The style conventions used in this document are detailed in Table 1.


[IV] Proprietary and Confidential Information of F5

Networks

Table 1: Conventions

Convention Use

Normal Text Bold Names of menus, commands, buttons, user-initiated CLI commands and

other elements of the user interface

Normal Text Italic Links to figures, tables, and sections in the document, as well as

references to other documents

Script Language scripts

Courier File names

Note: Notes which offer an additional explanation or a hint on how to

overcome a common problem

Warning: Warnings which indicate potentially damaging user operations and

explain how to avoid them


[V] Proprietary and Confidential Information of F5

Networks

Table of Contents

1. Introduction ..........................................................................................................................1

2. Upgrading Multiple SDC Sites Managed by an EMS Site to SDC 5.1 from SDC 4.4 ..........2 2.1 Working with the SDC during an Upgrade .............................................................................................4 2.2 Upgrading a Site ....................................................................................................................................5

Prerequisites ...................................................................................................................................6 Performing the Site Upgrade ........................................................................................................23 Post Site Upgrade Tasks................................................................................................................39

3. Upgrading a Standalone SDC Site to SDC 5.1 from SDC 4.4 ........................................... 46 3.1 Prerequisites ........................................................................................................................................46

Back Up Site Data .........................................................................................................................46 Back Up the /etc/udev/rules.d/70-persistent-net.rules File ........................................................46 Copy the SS7 License and Configuration Files ..............................................................................46 Map the Network Interfaces between Releases ..........................................................................47 Generate List of Bonds .................................................................................................................49 Update the Site Topology File.......................................................................................................49 Update the Installed Data Dictionary Version ..............................................................................57 Install the HP SSA CLI Utility .........................................................................................................58 Validate System Resources ...........................................................................................................58

Validate TC Timer Parameter .....................................................................................................60 3.2 Performing the Site Upgrade ...............................................................................................................60

Stop the Corosync Cluster ............................................................................................................60 Back Up Collected KPI Data ..........................................................................................................60 Perform a Split Mirror ..................................................................................................................61 Set up the Site Machines ..............................................................................................................62 Restart the EMS NMS Server ........................................................................................................66 Install the HP SSA CLI Utility .........................................................................................................66 Load the Backed Up Site Data ......................................................................................................66 Uploading the Site Topology File ..................................................................................................69 Installing the SDC Components ....................................................................................................72

Verifying the SDC Installation Status ..........................................................................................72 Generating ...............................................................................................................................72

3.3 Post Upgrade Tasks..............................................................................................................................74 Restore Backed Up Site Data ........................................................................................................74 Disable Session Data Replication by Tripo ....................................................................................75 Perform a Site Health Check .........................................................................................................76 Re-Mirror the Drives .....................................................................................................................76 Install the SS7 License ...................................................................................................................78 Add Licenses to New FEP IP Addresses.........................................................................................79 Modify Existing SNMP Configurations ..........................................................................................79 Reinstalling Data Center/Customer Environment RPMs ..............................................................79 Enable Wifi Offload.......................................................................................................................80

4. Rolling Upgrade to a New Release Version ..................................................................... 81 4.1 Prerequisites ........................................................................................................................................81

CLI Application ..............................................................................................................................81 New ISO Image .............................................................................................................................82


[VI] Proprietary and Confidential Information of F5

Networks

Pre-Upgrade Health Check ...........................................................................................................82 Updating the Salt Files ..................................................................................................................83 Setting the External Port Range....................................................................................................87

4.2 Uploading the New ISO image for the Component Upgrade ..............................................................87 4.3 Upgrading the Master Installer............................................................................................................88 4.4 Upgrading the OAM Database .............................................................................................................89 4.5 Upgrading the Tomcat Version ............................................................................................................90 4.6 Upgrading the Splunk Version .............................................................................................................90 4.7 Upgrading the SDC Components .........................................................................................................90 4.8 Restarting the API Flows Post-Upgrade ...............................................................................................91 4.9 Upgrading the Operating System ........................................................................................................92

Prerequisites for Upgrading the Operating System ......................................................................92 Uploading the New ISO Image ......................................................................................................93 Upgrading the Operating System .................................................................................................94

4.10 Monitoring the Rolling Upgrade Process ...........................................................................................95 Validating the SDC Processes .....................................................................................................95 Monitoring the Status of SDC Components................................................................................96 Monitoring SDC Release Versions ..............................................................................................96 Monitoring the Upgrade Logs .....................................................................................................96 Monitoring Salt Packages ...........................................................................................................96

5. Performing an Upgrade Rollback to 4.4 ........................................................................... 97 5.1 Rolling Back an SDC Site Upgrade to Release 4.4 ................................................................................97 5.2 Rolling Back an EMS Site Upgrade to Release 4.4 before Activating the Backup Array ....................100 5.3 Rolling Back an EMS Site Upgrade to Release 4.4 after Activating the Backup Array or a Standalone SDC Site Upgrade .....................................................................................................................................102

6. Performing an Upgrade Rollback to a Previous CF ....................................................... 104 6.1 Rolling Back an SDC Site Upgrade to a Previous CF ...........................................................................104 6.2 Rolling Back an EMS Site Upgrade to a Previous CF before Activating the Backup Array .................108 6.3 Rolling Back an EMS Site Upgrade to a Previous CF after Activating the Backup Array ....................109

Appendix A: Mapping the Interfaces between Releases .................................................. 111

Appendix B: Port Settings Used by the SDC ....................................................................... 113 B.1 EMS Site Internal Ports ......................................................................................................................113 B.2 EMS Site External Ports .....................................................................................................................114 B.3 SDC Site Internal Ports.......................................................................................................................117 B.4 SDC Site External Ports ......................................................................................................................119 B.5 HP Integrated Lights-Out (iLO) Port Settings .....................................................................................121

Glossary ................................................................................................................................ 122

List of Figures

Figure 1: GRUB Boot Loader Page ........................................................................................ 26

Figure 2: GRUB Boot Loader Page ........................................................................................ 63

Figure 3: Upgrade CLI Application Login .............................................................................. 82

Figure 4: Example of Removed Salt Files ............................................................................. 85


[VII] Proprietary and Confidential Information of F5

Networks

Figure 5: Example of Copied New Salt Files ......................................................................... 85

Figure 6: Example of Monitoring SDC Release Versions ..................................................... 96

List of Tables

Table 1: Conventions .............................................................................................................. IV

Table 2: High Level Structural Changes ................................................................................ 10

Table 3: “Networks” Element Level Changes ...................................................................... 11

Table 4: “Interfaces” Element Level Changes ...................................................................... 12

Table 5: “Route” Element Level Changes ............................................................................ 14

Table 6: “ApplicationInstances” Elements ........................................................................... 16

Table 7: “siteProperties” Element Level Changes ............................................................... 16

Table 8: Mandatory Parameters........................................................................................... 27

Table 9: Optional Parameters ............................................................................................... 27

Table 10: appStatus Command Error Codes ........................................................................ 37

Table 11: appStatus Return Codes ....................................................................................... 37

Table 12: High Level Structural Changes .............................................................................. 50

Table 13: “Networks” Element Level Changes .................................................................... 51

Table 14: “Interfaces” Element Level Changes.................................................................... 52

Table 15: “Route” Element Level Changes .......................................................................... 53

Table 16: “ApplicationInstances” Elements ......................................................................... 55

Table 17: “siteProperties” Element Level Changes ............................................................. 56

Table 18: Mandatory Parameters ........................................................................................ 64

Table 19: Optional Parameters ............................................................................................. 64

Table 20: appStatus Command Error Codes ........................................................................ 73

Table 21: appStatus Return Codes ....................................................................................... 73

Table 22: EMS Internal Ports .............................................................................................. 113

Table 23: EMS External Ports .............................................................................................. 114

Table 24: SDC Internal Ports ............................................................................................... 117

Table 25: SDC External Ports............................................................................................... 119

Table 26: HP iLO Ports ......................................................................................................... 121

Table 27: Common Terms ................................................................................................... 122

Table 28: Abbreviations ...................................................................................................... 123

F5 Signaling Delivery Controller

Bare Metal System Upgrade Guide

Introduction Working with the SDC during an Upgrade

[1] Proprietary and Confidential Information of F5 Networks

1. Introduction The F5® Traffix® Signaling Delivery Controller™ (SDC) 5.1 release can be upgraded to

from any SDC 4.4 release. In addition, SDC sites installed with an early build of SDC

release 5.1 can be upgraded to a later build of this SDC release.

The following procedures are described in this document:

▪ Upgrading Multiple SDC Sites Managed by an EMS Site to SDC 5.1 from SDC 4.4

▪ Upgrading a Standalone SDC Site to SDC 5.1 from SDC 4.4

▪ Rolling Upgrade to a New Release Version

▪ Performing an Upgrade Rollback to 4.4

▪ Performing an Upgrade Rollback to a Previous CF



Upgrading Multiple SDC Sites Managed by an EMS Site to SDC 5.1 from SDC 4.4 Working with the SDC during an Upgrade


2. Upgrading Multiple SDC Sites Managed by an EMS Site to SDC 5.1 from SDC 4.4

The SDC 5.1 release can be upgraded to from an existing SDC 4.4 release. This upgrade is

performed by upgrading each existing site individually. In a deployment with multiple SDC

sites managed by a central EMS site, the EMS site must be upgraded first. Once the EMS

site is upgraded, the SDC sites in the deployment should each be upgraded, in succession.

To upgrade multiple SDC sites managed by an EMS site:

Create an upgrade schedule, defining the order that the SDC sites will be

upgraded in after upgrading the EMS site.

Upgrade the EMS site, following the instructions in the Upgrading a Site

section.

Upgrade each of the managed SDC sites in succession, according to the order

that was defined. Make sure that each SDC site is successfully upgraded

before beginning the upgrade of the next SDC site.

Migrate all previously collected KPI data from an SDC site to the EMS site, by

performing the following steps on the SDC site:

i. Create a file of all collected KPI data by running the following commands

on the servers running the active NMS Agent(s) in the SDC site:

crm node standby

cd /opt/traffix/sdc/data/backup/statistics/

tar cvfz nmsagent.tar.gz nmsagent

ii. Copy the created SDC KPI data .tar file to the /data folder on the

EMS site by running the following command on one of the servers

running the OAMDB in the EMS site:





scp –rp nmsagent.tar.gz <IP_Address_of_OAMDB>:/data

iii. Run the following script on the EMS server that you copied the .tar file to,

making sure that the “site_id” value is the same as the value defined for

the SDC site name in the site topology file:

/opt/traffix/scripts/kpiMigration.sh -s <site_id>

iv. Verify that the following files were successfully created:

/data/old/statistics/<site_id>/upgrade_status.properties

Upgrade the SDC site, following the instructions in the Upgrading a Site section.

Warning: The EMS site upgrade cannot be rolled back once the SDC site upgrade

has begun.

Restore the collected KPI data from the SDC site to the EMS site by performing the

following steps:

i. Stop and start the NMS Agent in the EMS site by running the following

command on the server that the file was copied to:

monit stop <server_name><nmsagent_name>

monit start <server_name><nmsagent_name>

ii. Verify that the restore is completed by checking in the

/data/old/statistics/<site_id>/upgrade_status

.properties file for the following message:

upgrade_status=upgrade_done

iii. Delete the .tar file(s) from the /data folder.

Repeat this step until all SDC sites are upgraded.





Warning: The previous upgrade of any SDC sites cannot be rolled back once the

SDC site upgrade has begun.

Remove the additional VIPs that were configured to support communication

between the SDC and EMS sites during the upgrade:

Run the following command on one of the Installer servers in the EMS site:

Note: Perform this step per additional VIP to remove each additional VIP

separately. In the event that the VIPs are not removed, there may be a problem with

connecting to the upgraded VIPs, as the ARP (Address Resolution) Table will not be

able to resolve the IP addresses to the MAC addresses as there are different MAC

addresses in Release 4.4 and Release 5.1. See Align the MAC Addresses, in the Post

Site Upgrade tasks to resolve this potential problem.

Warning: Verify that all preceding steps have been successfully completed, and all

EMS and SDC sites have been upgraded, before performing this step.

python /srv/traffix/pillar/remove_ip_from_oamdb.py <Server_Name>

<Interface_Name>

Run the following command on one of the Installer servers in the EMS site:

salt '*' state.highstate

2.1 Working with the SDC during an Upgrade While upgrading multiple SDC sites that are managed by an EMS site, an upgraded EMS

site can manage SDC sites that are running with SDC release 4.4.

However, the following features, are disabled until all SDC sites are upgraded. Therefore,

these features should not be configured using the EMS Web UI until all SDC sites have

been upgraded:



Upgrading Multiple SDC Sites Managed by an EMS Site to SDC 5.1 from SDC 4.4 Upgrading a Site


▪ Broadcast Pool

▪ Notification Pool

▪ Flows table

In addition, the following configurations are only reflected in the upgraded EMS site and

not in the SDC sites:

▪ SNMP server

Note: While in mix-mode, the EMS Alarms, the EMS Dashboard, and the EMS Backup

and Restore (Administration) Web UI do not display the alarms, reports (except for Splunk

and TDRs) and backups, respectively, for its related SDC sites. Once the SDC site is also

upgrade to 5.1, these EMS Web UI screens will reflect the SDC site alarms, reports, and

backups as expected.

2.2 Upgrading a Site The SDC 5.1 release can be upgraded to from an existing SDC 4.4 release. This upgrade is

performed by upgrading each existing site individually.

This section describes the steps necessary to upgrade an SDC or EMS site. Note that certain

steps are only applicable when upgrading an EMS or an SDC site. These steps are clearly

marked.

In a deployment with multiple SDC sites managed by a central EMS site, the EMS site

must be upgraded first. Once the EMS site is upgraded, the SDC sites in the deployment

should each be upgraded, in succession. See Upgrading Multiple SDC Sites Managed by

an EMS Site to SDC 5.1 from SDC 4.4 for more information and details about upgrading

this kind of deployment.

Note: When upgrading a site, always upgrade the master Installer servers first.





Note: When upgrading a standalone SDC site, follow the procedure described in

Upgrading a Standalone SDC Site to SDC 5.1 from SDC 4.4.

This section includes the following steps:

▪ Prerequisites

▪ Performing the Site Upgrade

▪ Post Site Upgrade Tasks

Prerequisites The following prerequisites must be completed before starting the upgrade:

Back Up Site Data It is recommended to create an external backup of the EMS site data folder before

beginning the upgrade.

Back Up the /etc/udev/rules.d/70-persistent-net.rules File The /etc/udev/rules.d/70-persistent-net.rules file contains the site

networking definitions. These definitions may be modified during the upgrade and the

backup file will be needed to support upgrade rollbacks. Create a backup copy of this file

before beginning the upgrade.

Copy the SS7 License and Configuration Files

Note: This is only relevant for SDC sites that are configured to work with SS7.

After the upgrade, the SS7 license must be installed on the upgraded site servers. Before

beginning the upgrade, copy the SS7 license for easy access after the upgrade. In addition,

the SS7 configuration files must be copied to each upgraded SDC site that has an SS7

driver with a CPF.





To copy the SS7 license:

Go to the /opt/DSI/ directory.

Copy the SS7 licenses file - <filename>.lic to a local directory.

To copy the SS7 configuration files:

1. Copy the following files to a local directory:

▪ /opt/DSI/config.txt and /opt/DSI/system.

▪ /opt/traffix/sdc/config/ss7/routing/e212Toe214-v2.csv

▪ /opt/traffix/sdc/config/ss7/routing/mmeToHlr.csv

▪ /opt/traffix/sdc/config/ss7/routing/hss.csv

▪ /opt/traffix/sdc/config/ss7/routing/hlr.csv

Note: If a new version of the SS7 driver is installed during the upgrade, compare the

original configuration files to the new version files. If configuration file sets are not

identical, ensure that the relevant SS7 driver version files are later restored during the

upgrade

Map the Network Interfaces between Releases The networking definitions previously defined in the SDC 4.4 release may be modified for

the SDC 5.1 release. These modifications must be reflected in the definitions configured in

the site topology file for any EMS and/or SDC site before upgrading.

This section describes how to create a table of the networking changes. For more

information about how to update the site topology file, see Update the Site Topology File.





To map the networking between SDC 4.4 and SDC 5.1:

Extract the 5.1 release ISO image to a local folder and go to

traffix/repositories/traffix folder.

Locate the salt-srv.rpm package, and extract the data to a local folder.

Go to the scripts folder located at the following path:

..\<customer_defined_local_folder>\salt-

srv<latest_version>.noarch\salt-

srv<latest_version>.noarch.cpio\.\opt\traffix\salt-

srv\<latest_version>\archive.tar\salt\system\scripts

\

Copy this folder to each site server, and run the following commands:

dos2unix showNetMap.sh

chmod +x showNetMap.sh

Run the following script:

./ showNetMap.sh

As shown in the following example output, a table is displayed with the current network

interfaces and the corresponding new network interfaces that will be used by the upgraded

server:

CURRENT NEW HWADDR

eth8 eth0 ac:16:2d:b9:1b:f0










eth2 eth6 ac:16:2d:7d:11:48

eth3 eth7 ac:16:2d:7d:11:4c

eth10 eth8 ac:16:2d:7d:11:49

eth11 eth9 ac:16:2d:7d:11:4d

eth12 eth10 ac:16:2d:7d:11:4a

eth13 eth11 ac:16:2d:7d:11:4e

Generate List of Bonds Using the following command, generate the list of bonds that were used by the SDC 4.4

site. These bonds must be used – without modifications – in the site topology file for SDC

5.1.

To generate the list of bonds used by the SDC 4.4 site:

Run the following command:

grep "MII Status\|Duplex" /proc/net/bonding/bond*

Update the Site Topology File The SDC 5.1 release includes a modified site topology file structure. To ensure all statistics

and configuration data are correctly migrated during an upgrade from previous SDC

releases, it is important to understand the current file structure and verify that the existing

definitions for the hostnames, networking configurations (including bonds), and

component names are maintained in the new site topology file.

The high level structural changes between the site topology file structure for SDC release

5.1 and previous releases are detailed in Table 2.





Table 2: High Level Structural Changes

Previous Element Name New Element Name in SDC 5.1 Comments

Hosts Vms

Bonds - The bonds are defined for each

virtual machine under a specific

“vm” element in the “vms”

element.

Interfaces - The mapping of each mac to an

interface may change in SDC

5.1

Note: If a new version of the

SS7 driver is installed during the

upgrade, compare the original

configuration files to the new

version files. If configuration file

sets are not identical, ensure that

the relevant SS7 driver version

files are later restored during the

upgrade

Map the Network Interfaces

between Releases.

Cluster - In SDC 5.1, the SDC

components (applications) are

defined as application instances

per VM, and not as a cluster of

components.

- applications

Networks Networks






Routerules - Not used in SDC 5.1

Routes - The route information is defined

for each virtual machine under a

specific “vm” element in the

“vms” element.

Properties Siteproperties

processes - Not used in SDC 5.1

To create a site topology file to upgrade a site from a previous SDC release to SDC

5.1:

Create a site topology file for an SDC 5.1 site, following the guidelines in the

F5 SDC Guidelines for Creating a Site Topology File.

Under the “networks” element, verify that the previously configured values are defined

under the correct element in the SDC 5.1 site topology file. Table 3 details the relevant

element names and the corresponding name in SDC 5.1.

Table 3: “Networks” Element Level Changes


Id Name

Vlan Vlan

Net net4

ip4sub ip4sub

net6 net6

ip6sub ip6sub

Id Role The valid values for this field

are:






▪ ic

▪ mgmt

▪ sig

▪ custom

Under the “vms” element, verify the following:

There is a “vm” element for each of the “host” elements in the previous topology

file, and the “name” values have not changed.

When defining the vms for an EMS site, verify that the “Issplunkmaster” element

is only set to “true” on the first vm in the site.

The “defaultgateway” element value has not changed, and is defined for each

“vm” element.

Under the “interfaces” element, verify that the previously configured values

are defined under the correct element in the SDC 5.1 site topology file. Table

4 details the relevant element names and the corresponding name in SDC

5.1.

Note: In previous releases, these values were configured under the “vlans”

element.

Table 4: “Interfaces” Element Level Changes


Name network Verify that the value matches the

previously defined value under

the “networks” element.

Id name






Net - This is already defined under the

“networks” element.

net6 - This is already defined under the


ip4 ip4

ip4Sub - This is already defined under the


ip6 ip6



Interface Dev

- bondDev Verify that the value matches the

defined element for the

interfaces value for the

corresponding bond element in

the previous release.

- bondingOpts Verify that the value matches the



corresponding bond_option and

mode elements in the previous

release.

Under the “route” element, verify that the previously configured values are

defined under the correct element in the SDC 5.1 site topology file. Table 5

details the relevant element names and the corresponding name in SDC 5.1.





Note: In previous releases, these values were configured under the “route”

element.

Note: A route can be defined individually for IPv4 or IPv6, but not for both.

Note: Only a route defined as part of an interface that is assigned to a VIP

application will be defined with a source IP.

Table 5: “Route” Element Level Changes


Id name

Destnet net4 If this was not defined in the

previous release, enter the value

that was previously defined for

the “net” element under the

network with an id that matches

the name of this route.







destmask ip4sub Convert the destmask to CIDR.

For example, 255.255.255.1 is

24.

If this was not defined in the



the “ip4sub” element under the










24.







Gateway gateway

Fep -

Nic -

src_ip -

target -

sourcenet -

Under each “applicationinstances” element, define the applications that run on

the vm. These applications were previously defined as part of the SDC

cluster. Table 6 details the element names in SDC 5.1.

Note: Application instances must be defined on the same servers (vms/hosts)

as they were in previous SDC releases. In SDC 5.1, all applications run in active-

active mode. The FEP application is the only application that supports active-

standby mode, and this support is provided by defining a VIP application

instance for each FEP instance.





Table 6: “ApplicationInstances” Elements

Element Name in SDC 5.1 Comments

Type

Name Names are taken from the instance attribute for the resource

element of type app for the application group in the Cluster

element.

Note: If the instance value is empty, the following components

must be named as such:

▪ CPF - cpf1

▪ NMS Agent - nmsagent1

▪ Configuration Manager - traffix_config_mgr-config1

▪ Web UI - webui1

▪ Tripo – tripo1

Other components can be defined with any name.

listenInterfaceName The name of the network interface (s) used by the application.

This must be defined for each application.

Under the “siteProperties” element, verify that the previously configured values are

defined under the correct element in the SDC 5.1 site topology file. Table 7 details the

relevant element names and the corresponding name in SDC 5.1.

Note: In previous releases, these values were configured under the “properties”

element.

Table 7: “siteProperties” Element Level Changes


SITE_ID Name

- sdcVersion

Timezone Timezone






ntpServers ntpServers Define each ntpServer as a sub-

element (“ntpServer”), not as a

string.

traffixFolder traffixFolder

- isManager Define this value as “true” for

an EMS site.

isMultiSiteEnv isMultiSiteEnv Define this value as “true” if the

deployment includes SDC sites

that are managed by an EMS

site.

- splunkAvailable Define this value as “true” if the



site.

EmsServer1

EmsServer2

emsServers Define each emsServer as a sub-

element (“emsServer”), not as a

string.

Note: For SDC site topology files

only.

Update the Installed Data Dictionary Version Release 5.1 includes an updated data dictionary. When upgrading existing systems,

however, this data dictionary is not installed, and the previous data dictionary is kept.

Therefore, to ensure that the updated data dictionary version is installed in the upgraded

site, it must be manually updated to the 4.4 site before the upgrade.





Note: This prerequisite is only relevant for SDC sites installed with release 4.4 CF 18

and earlier.

To manually update the installed data dictionary version:

Download and unzip the SDC release 5.1 package.

Go to the \opt\traffix\<SDC_release_version>\config\diction

aries folder, and save the superDictionary.xml file locally.

If any manual changes were made to the installed data dictionary, copy those

changes to the saved superDictionary.xml file.

Using the SDC 4.4 Web UI, replace the installed data dictionary with the

updated .xml file, by performing the following steps:

Go to Routing > Data Dictionary. The Data Dictionary screen displays the

currently selected data dictionary.

Click Browse and select the data dictionary file’s location.

In the Protocol field, select the data dictionary’s supported protocol.

Click Submit. The newly selected data dictionary appears in data dictionary list.

Verify Session Data Replication by Tripo The session data saved in the Tripo (session repository) is replicated between session

repositories on geo-redundant SDC sites. In previous SDC releases, this replication was

performed by default by the CPF and FEP components on the two sites. In the SDC 5.1,

this replication is performed by default by the Tripo (session repository) itself. Therefore,

before upgrading from SDC 4.4, session data replication performed by the CPF and FEP

components must be disabled, and session data replication performed by the Tripo (session

repository) must be enabled.





To configure Tripo-based session data replication:

Enable Tripo-based session data replication by running the following

commands on each server running Tripo Resource:

su - traffix

cd /home/traffix/Tripo/env/linux-x86_64/

. DefEnv Tripo

UI_Config

set SiteReplication true

CTRL–C to exit from UI_config

UI_Config -w ConfigParams.cfg

Verify that Tripo-based session data replication was enabled by running the following commands:

Run UI_config

==>dump

Review the ConfigParams.cfg file (located in the /home/traffix/Tripo/cfg/ folder)

and verify that SiteReplication = true

Disable CPF and FEP-based session data replication on each server that has a CPF or FEP resource on both SDC sites:

Go to the following folder: /opt/traffix/sdc/config/sysconfig/

Within the folder, open the “traffix” file.

Add the parameter USE_TRIPO_REPLICATION=true. If the parameter exists,

change the setting from false to true.

Save the file and restart each CPF and FEP resource one by one.





Install the HP SSA CLI Utility The HP SSA CLI Utility is used throughout the upgrade.

To install the HP SSA CLI Utility RPM file (hpssacli.rpm):

Download and install the HP SSA CLI Utility RPM file (hpssacli.rpm) from

the following link:

▪ For RHEL 6:

http://downloads.linux.hpe.com/repo/spp/redhat/6/x86_64/current/

▪ For RHEL 7:


Validate System Resources An integral part of the upgrade process is the site mirror process. This process designates

one of the HP Arrays as a backup array, ensuring support for upgrade rollbacks. The split

mirror process is supported by a specific HP configuration.

Just before beginning the upgrade, perform the following steps to validate that your HP

configuration is supported:

Note: Perform this prerequisite as close as possible to performing the upgrade.

Validate the RAID Controller type by running the following command:

hpssacli ctrl all show

The supported controller types are:

▪ BL460 gen8, with P220i controller

▪ BL460 gen9 with P244br controller

▪ DL380 gen8 with P420i (+1GB FBWC)







RAID1 / Mirror

RAID 1+0 (Mirror+Stripe, 4 drives)

▪ DL380 gen9 with P440ar (2GB FBWC)

Validate the RAID Controller cache size by running the following command:

hpssacli ctrl all show detail |grep -i “Total Cache Size”

The minimum supported cache size is:

▪ 512MB (Blade)

▪ 1GB FBWC Cache (DL380)

Verify that the RAID Controller has battery backup by running the following command

and getting an OK response:

hpssacli ctrl all show detail |grep -i Battery |grep -i Status

Validate the RAID Controller firmware version by running the following command:

hpssacli ctrl all show detail |grep -i Firmware

The minimum supported firmware version is:

▪ RAID Adapter Firmware Version:

▪ 5.42 and later

▪ 4.68

Stop Traffic to the Site

Note: This prerequisite is applicable only when upgrading SDC sites. Perform this

prerequisite as close as possible to performing the upgrade.

When upgrading the SDC site, all site servers will be down. Therefore, before beginning

the upgrade, verify that all expected traffic is routed to an alternate SDC site.





Configuring the SDC Site Ports When upgrading from Release 4.x to 5.1 and the deployment is in mix-mode (SDC local

sites remain in 4.x, while the EMS site is already upgraded to 5.1), you need to manually

configure the SDC local site NMS Agent and configuration manager ports, so that the EMS

can successfully connect to them with the default port numbers.

Note: This procedure is only relevant if there was a previous upgrade within release

4.x where the NMS port was changed from, its default value, 10040, to 10050 and the

configuration manager port was changed from, its default value, 16167 to 16169. Verify

the port values to see if they were changed from their default values.

To reconfigure the ports:

Note: This procedure must be performed on each server that hosts the

configuration manager and NMS Agent.

Stop the configuration manager and NMS Agent.

Edit the following files in the 4.x installation:

▪ config/sysconfig/traffix

▪ config/sysconfig/traffix_nms*

▪ config/sysconfig/traffix_config_mgr

Change the ports as follows:

For the configuration manager, change to 161617

For the NMS Agent, change to 100040.

Note: The port values appear several times throughout the config files.





Re-start the configuration manager and NMS Agent.

Repeat this procedure for all relevant servers.

Verify flowManager XML File Size Verify that the flowManager XML File size is the same on the 4.4 EMS and SDC sites. In

the event that the file sizes are different, some objects may be deleted post-upgrade.

The file is located: /opt/traffix/<sdc-4.4. release version>/data/backup/<Site / EMS name>

/<configuration manager instance name>/flowManager/

For example, /opt/traffix/sdc-4.4.7-1/data/backup/KC/kscymo-f5dra01-c7000-

01_traffix_config_mgr-config1/flowManager/

Validate TC Timer Parameter The TC Timer parameter (Topology > Specific Site Settings > Site> SDC Components

or Topology > Remote Peers) in 4.4 cannot be defined as more than 30,000 milliseconds.

If it is, then the Reestablish Connection Time parameter in 5.1 cannot be validated and any

other edits to the SDC Component Properties cannot be saved.

Performing the Site Upgrade

Note: All configuration changes performed locally on SDC sites while the EMS site is

being upgraded will be overridden by the configuration data distributed by the EMS site

once it has been upgraded. It is therefore recommended to not perform any configuration

changes on the SDC site(s) while the EMS site is being upgraded.

Performing a Split Mirror During this phase, a backup array is created to enable upgrade rollbacks.





To create a backup array:

Run the following command on each site server to switch the servers to

standby mode.

crm node standby

Perform the Split Mirror procedure to designate one of the HP Arrays as a backup array:

Run the following command to verify that the logical drive is in an “OK” state:

hpssacli controller slot=0 ld all show

Run the following command to create a backup array:

hpssacli controller slot=0 array A splitmirror action=splitwithbackup

Enter “y” to confirm that you want to continue after being prompted with the

warning message.

Run the following command to verify that there are two arrays, each with a logical

drive, and that backup logical drive was created.


Note: To ensure successful rollbacks, be sure to clearly mark which array and

logical disk are the backup.

Run the following command to create a visual marker for the mirrored array:

hpssacli controller slot=0 <active_array_ID> modify led=on

Setting up the Site Machines In this phase, the site machines are rebooted and the master and minion servers are created.





Note: Always set up the master Installer servers and upload the site topology file to

them before setting up the other site machines.

You need to install the operating system on each site machine. The operating system is

installed from the ISO image.

Note: You can load the ISO image using the ILO Integrated Remote Console, any other

available console or directly from your system.

2.2.2.2.1 Installing the Operating System In order to install the operating system, you need to load the ISO image.

Note: The ISO image must be loaded for each server in the site.

To load the ISO image:

Note: For an EMS deployment, the following steps can be run in parallel on each EMS

site.

Select the ISO image from where it is saved on your computer.

Start the installed site machine from the ISO image.

The Welcome To F5 Traffix SDC Install Menu is displayed.

Under the Welcome To F5 Traffix SDC Install Menu, select Install Traffix F5 EL

from cdrom for bare metal.

The GRUB boot loader page displays.





Figure 1: GRUB Boot Loader Page

2.2.2.2.2 Defining Master and Minion Servers The GRUB boot parameters define a server's role as either a master Installer server or as a

minion server that will host the SDC components. Configuring the parameters is done from

the GRUB boot loader page. There are mandatory parameters and optional parameters that

are only required if relevant for the deployment.

To configure the GRUB boot parameters:

In the prompt line, after F5-TRAFFIX_SDC:traffix/kickstart/kickstart.cfg,

press the TAB key to enable editing and add the parameters, as follows:





Table 8: Mandatory Parameters

Name Value Description

server master/minion

hostname the server's hostname

Note: The hostname must be identical (case sensitive) to the value

defined under the name attribute for the vm element in the Site

Topology file

master0 The IP address on the management network that the first vInstaller

uses.

master1 The IP address on the management network that the second

vInstaller uses.

ip The IP address is from the management network interface for

minion and master Installer servers

netmask netmask for the IP address defined above (CIDR is not supported

on IPv4)

device The ethernet interface used by the IP address defined above

Table 9: Optional Parameters


vlan vlan number for interface(if vlan defined)

gw default gateway (need to be mandatory if server = master)

debug debug=yes enable salt log with debug

dns DNS

The following is an example of inputted GRUB parameters:





Click ENTER when done entering the parameters.

Repeat these steps for each site machine.

Each site server is now installed with an Operating System and has a defined role

(master or minion). You can verify which servers in a site are defined as a master

or minion with the siteStatus API Request.

2.2.2.2.3 Modifying the GRUB Boot Parameters The GRUB boot parameters are saved in the params file. If you want to change any of the

parameters or add a parameter, you can do so by editing the params file.

Note: You can only edit the parameters at this stage in the installation process, prior to

uploading the Site Topology file. After editing any GRUB boot parameters in params File,

you must run an installation script.

You can reconfigure a master server as a minion server, but you cannot reconfigure a

minion server as a master server.

To edit the GRUB boot parameters:

Go to the directory where the params file is saved:

# cd /var/tmp/salt-install/

Execute the following command to edit the relevant parameters:

vi params





Note: If when entering the GRUB parameters from the GRUB boot screen, you

clicked ENTER, but you still want to edit the parameters, you need to use the example

params file to edit by executing the following command:

cp params.example params

Add/Edit a parameter according to the list of mandatory or optional parameters (see

Table 8 and Table 9).

Execute the following command to run the installation script:

./salt-install.sh

Each installed site server is now updated with the edited parameters. You can

verify which servers in a site are defined as a master or minion with the siteStatus

API Request.

Note: If you changed the IP address of the master Installer or any of the GRUB

parameters in the params file, then you need to run the installation script on all of the

minion servers. If the master Installer IP address has not been changed, then you need

to restart the Salt minion service.

2.2.2.2.4 Verifying the Site Networking Definitions The site networking definitions in the /etc/udev/rules.d/70-persistent-

net.rules file may have changed while installed the new operating system. Compare

this file to the saved backed up file, and edit the interface configurations in the site topology

file based on the updated definitions.

Loading the Backed Up Site Data Once the EMS site servers are successfully created, the backed up site data is loaded from

the logical drive previously defined as the backup logical drive.





Note: This step is not relevant for SDC site upgrades. Once the SDC site is upgraded,

the EMS site will distribute this data to the SDC site.

Note: This should be performed on each EMS site server.

Install the HP SSA CLI Utility RPM file (hpssacli.rpm) by running the

following command:

yum install hpssacli

Migrate the backed up site data saved in the Config Manager:

Enable access to the backup disk created with the Split Mirror mechanism:

iv. Run the following command on the active array on each EMS machine:

hpssacli controller slot=0 <active_array_ID> splitmirror

action=activatebackup

Warning: Potential loss of site data!

Once this step has been performed, the logical drive is no longer defined as a backup

drive, and its data is no longer protected from being overwritten. Any ISO installation

performed after this point will result in the data on both logical drives being

overwritten. To ensure that the data on the backed up logical drive will not be lost,

follow the rollback procedure in the Rolling Back an EMS Site Upgrade to Release

4.4 after Activating the Backup Array to return the site servers to their pre-upgrade

state (running SDC 4.4) before starting the upgrade from the beginning.

Run the following command to ensure that the logical drive that is used is the logical

drive that is being upgraded and is running with SDC release 5.1:





hpssacli controller slot=1 ld <ID of the active Logical Drive> modify

bootvolume=primary

Run the following commands on both EMS machines to identify the activated disk

that was previously backed up:

grep device /etc/lvm/backup/vg1 -B1 | grep id

pvs –o pv_name,vg_name,pv_uuid

The output of the first command includes the disk that the system recognizes,

while the output of the second command includes a list of all the disks that are

installed in the system.

Identify the disk in the output of the second command that did not appear in the list

of recognized disks (in the output of the first command), and run the following

command using the disk name of the disk that did not appear in the list of

recognized disks:

vgimportclone --basevgname vg1_clone /dev/sd<name of disk>

Run the following commands to mount and activate the volume that you want to

restore data from:

lvchange -a y vg1_clone/lv_opt

mount -t ext4 -o ro /dev/mapper/vg1_clone-lv_opt /mnt

Note: If the site must be re-installed after this stage, perform a rollback. For more

information, see Performing an Upgrade Rollback.

Verify that the /data folder (or its subdirectories) is available for data migration:





i. Locate the /data folder (or its subdirectories) in the site topology file and

verify that it has been defined as a “mountPoint” value, meaning, to be

used as a volume. Once this has been verified:

ii. Copy the updated site topology file to each site machine.

iii. On each EMS server, run the emsMigration.sh script using the –m

argument, followed by the full path to the topology file:

/srv/salt/<current version>/system/scripts/emsMigration.sh -m

<path_to_topology_file.xml>

Copy the previous backup folder from the backup disk to the verified data folder:

i. On each EMS server, run the following command:

cd /mnt/traffix/<sdc_version>/data

cp –rp backup /data/

Note: After loading the backed up site data, ensure that in the case of a reboot the

active array will be rebooted as the primary array by running the following command:

hpssacli controller slot=<relevant slot ID> ld

<active_array_logical_drive_number> modify bootvolume=primary

hpssacli controller slot==<relevant slot ID> ld

<backup_array_logical_drive_number> modify bootvolume=secondary

hpssacli controller slot==<relevant slot ID> ld 3 modify bootvolume=none

Uploading the Site Topology File In this phase, the site topology file is uploaded to the master Installer servers using an API

request. Before executing the API request to upload the Site Topology file, you need to

validate the Site Topology File and then upload it.





Note: Upload the Site Topology file only after the master Installer servers are up and

running.

To validate the Site Topology File:

Run the following command from cd /srv/traffix:

#python pillar/traffix_validate.py /tmp/ topology.xml

The following is an example of a successful validation:

validate_topology_string: succeeded to validate xml file topology=<?xml

version="1.0" ?>

The following is an example of a failed validation:

validate_topology_string: failed to validate xml file topology=<?xml

version="1.0" ?>

To upload the Site Topology file to the Installer Machines:

Upload the Site Topology file to a master Installer server.

Authenticate the Installer REST interface by sending the following request to

the master Installer to generate an authentication token:

Note: An authentication token expires after ten hours.

curl -ksi https://<master_IP_address>:8000/login -H "Accept:

application/json" -d username='saltuser' -d password='traffix' -d

eauth='pam'

Note: For all API requests, you need to use the minus sign, for example "-d" and

not the N-dash "-". If you copy–paste the API request, you may have to type in the "-

d" again with the minus sign to avoid syntax conversion errors.





The success return code for this request is ‘200’.

After validating the Site Topology file and generating an authentication token, run the

following API command to upload the topology file:

curl -ksi https://<master_IP_address>:8000 -H "Accept: application/x-

yaml" -H "X-Auth-Token: <Token>" -d client="runner" -d

fun="traffix.uploadTopology" -X POST -d @topology.xml

The response indicates if the Site Topology file has been successfully uploaded. The

following is an example of the API request with a successful response:

# curl -ksi https://localhost:8000 -H "Accept: application/x-yaml" -H

"X-Auth-Token:aa6790aa67ae5ce87715b66bf5bd58fc3ea4bdb5" -d

client="runner" -d fun="traffix.uploadTopology" -X POST -d

@topology.xml

HTTP/1.1 100 Continue

HTTP/1.1 200 OK

Content-Length: 63

Access-Control-Expose-Headers: GET, POST

Access-Control-Allow-Credentials: true

Vary: Accept-Encoding

Server: CherryPy/3.2.2

Allow: GET, HEAD, POST

Cache-Control: private

Date: Tue, 15 Dec 2015 13:27:04 GMT

Access-Control-Allow-Origin: *

Content-Type: application/x-yaml

Set-Cookie: session_id=6db032df27701c9dfa9dbbf9d8a8a42c682dbb68;

expires=Tue, 15 Dec 2015 23:27:04 GMT; Path=/

return:

- - 0





- topology uploaded to the server successfully

The Site Topology parameters are now saved in the Cassandra database in both of the

master Installers. Once the Site Topology file has been uploaded successfully to the

master Installer servers, set up the remaining site machines by following the Setting

up the Site Machines procedure.

Note: Once the Site Topology file has been uploaded successfully, the only way

to modify the site configurations (in the params file or in the Site Topology file) is to

perform a new installation by reinstalling the ISO.

Installing the SDC Components The minion servers communicate with the master Installer servers and the master Installer

servers then reply to the minion servers, based on the Site Topology definitions, to know

where to install the different SDC components (FEP, CPF, etc.).

2.2.2.5.1 Verifying the SDC Installation Status This verification is done by invoking the “appStatus” REST API. Applying this REST

APIs queries the master Installer about the status of the SDC applications running on a

specific server or on all the servers. This API is based on a standard Salt API interface and

the body of the REST API message contains CLI Salt functions.

The site is only successfully installed once the “12000” result code is returned for the

appStatus API request.

2.2.2.5.2 Generating an Authentication Token Before any REST API request can be sent, you must have a valid authentication token. You

need to send a request to the master Installer to generate an authentication token.






To generate an authentication token:

Send the following API request to the Master Installer that is identified by the

<master_IP_address> parameter:



eauth='pam'

Note: For all API requests, you need to use the minus sign, for example "-d" and not

the N-dash "-". If you copy–paste the API request, you may have to type in the "-d" again

with the minus sign to avoid syntax conversion errors.

2.2.2.5.3 Authentication Request Status Codes The following are the possible return codes for the authentication API request:

Return Code Description

200 success

401 authentication required

406 requested Content-Type not available

2.2.2.5.4 Application Status per Server This API request checks the status of a specific server. The response includes the relevant

status codes for successfully installed applications. In addition, as with all other API

requests, there are related command execution codes.

2.2.2.5.5 appStatus API Request curl -ksi https://<master_IP_address>:8000 -H "Accept: application/x-

yaml" -H "X-Auth-Token:<Token>" -d client="runner" -d

fun="traffix.appStatus" -d tgt="*" -d apps=True (optional for apps

list)





2.2.2.5.6 Command Execution Codes for appStatus API Request Table 10: appStatus Command Error Codes

Exit Code Description

-50 Failed to validate site topology file - check site topology file

-51 Installation not started yet

-52 Could not get information from DB

2.2.2.5.7 Return Codes for appStatus API Request Table 11: appStatus Return Codes


14002 Pending Machine Start

14003 Pending SDC Installation

14004 Pending SDC Start

14006 Pending SDC Stop

15002 Fail VM Start

15003 Fail To Install SDC

15004 Fail To Start SDC

15006 Failed To Stop SDC

13000 Suspended

12000 Successfully installed

Migrating Splunk Data In this phase, the backed up Splunk data is uploaded to the upgraded servers.

Note: This step is only performed when upgrading EMS sites.





To migrate backed up Splunk data:

Stop the Splunk processes on both EMS machines:

Run the following command to check if any Splunk processes are running:

monit summary

Run the following command to stop any Splunk processes that are running

(Splunksearch, splunkforwarder, Splunk, SplunkMaster):

monit stop <process_name>

Run the following command to verify that all Splunk processes are stopped:

monit summary

Migrate the backup folder to the new location:

On each EMS server, run the emsMigration.sh script using the –s argument,

followed by the full path to the location where the previous Splunk directories are

saved:

/srv/salt/<current version>/system/scripts/emsMigration.sh -s /mnt/

Start the Splunk processes:

Run the following command to check if any Splunk processes are running:

monit summary

Run the following command to start any Splunk processes that are not running:

monit start <process_name>

Run the following command to verify that all Splunk processes are running:

monit summary





Verifying Site Component Status To verify the site component status:

Verify that all the EMS site components are up and running according to the

site topology file by running the following command:

monit summary

Note: If the site components are not installed, up, and running, as expected,

perform an upgrade rollback. For more information, see Performing an Upgrade

Rollback.

Post Site Upgrade Tasks

Align the MAC Addresses In the event that the VIPs were not removed, you need to update the ARP (Address

Resolution) Table in the switch to align the different MAC addresses in Release 4.4 and

Release 5.1.

To update the ARP Table:

Send a ping from each interface: ping -I eth1 google.com

Perform a Site Health Check Verify that the site is running as expected by performing a health check:


/srv/traffix/upgrade/upgrade-cli

Log in to the upgrade CLI, using traffix as your username and password.

Note: If you have changed your Web UI password since the upgrade, use the new

password to log in to the upgrade CLI.





Run the following command in the upgrade CLI:

healthcheck

If site components are up and running as expected, the EMS site has been

successfully upgraded. If not, perform an upgrade rollback. For more


Re-Mirror the Drives After verifying that the site is up and running as expected, re-mirror the drives by

performing the following procedure, based on the site that was upgraded:

▪ After upgrading an EMS site, re-mirror the drives by performing the Re-Mirroring

the Drives on an EMS Site procedure.

▪ After upgrading an SDC site, re-mirror the drives by performing the Re-Mirroring the

Drives on an SDC Site procedure.

Re-Mirroring the Drives on an EMS Site

Warning: Performing this procedure removes the option to roll back the upgrade on the

SDC or EMS site.

To re-mirror the drives on an EMS site:

Access the HP Storage Controller Administrator BIOS Interface by performing

the following steps:

Enter HP SSA in the remote console.

Reboot the server.

At the prompt, enter HP Storage Controller Administrator.

Press F5 (for Gen8) or F10 (for Gen9) to start the HP Storage Controller

Administrator.





Define and activate the active array as the master array by performing the following

steps:

Under Array Controller(s) select the backup array

From Actions list - click Configure.

Under Controller Devices, select Arrays.

Under Arrays, select the active array.

Under Actions select Re-Mirror Array.

Re-Mirroring the Drives on an SDC Site

Warning: Performing this procedure removes the option to roll back the upgrade on the

SDC or EMS site.

To re-mirror the drives on an SDC site:

Run the following command on the active array on each SDC machine:

hpssacli controller slot=0 <active_array_ID> splitmirror action=remirror

Run the following command to check the progress of the re-mirroring:


The output indicates the percentage that has been re-mirrored, for example:

Smart Array P220i in Slot 0 (Embedded)

array A

logicaldrive 1 (279.4 GB, RAID 1, Recovering, 23% complete)

The re-mirror is complete once the output shows the logical drive status as “OK”.





Install the SS7 License


Each server running SS7 must have a license installed.

To install the SS7 license, perform the following step on every server that will run

SS7:

Copy the saved SS7 license file to /opt/DSI

Stop and start the CPF component by running the following:

monit stop <server_name><cpf_name>

monit start <server_name><cpf_name>

Add Licenses to New FEP IP Addresses Each FEP IP address must have a license. During the upgrade, additional IP addresses may

have been added to the FEP instances. If additional IP addresses were added, these IP

addresses much each have their own license. For more information about obtaining the

license, contact F5 Support and refer to the F5 SDC User Guide on how to add a new

license key.

Modify Existing SNMP Configurations The system alarms and KPIs were modified in release 5.1. Therefore, the previous SNMP

configurations and related scripts must be adapted according to the 5.1 MIB file. Verify

that the SNMP configurations have been updated according to the MIB file included in

release 5.1. As of release 5.1 CF 9, all upgraded sites, will be enabled with SNMPv3

functionality. For more information about SNMPv3, refer to the 5.1 User Guide.

Note: While in mix-mode, you cannot create or update SNMPv3 users or their security

settings, even on the site that is already upgraded. Only after the upgrade is completed, and





all sites have been upgraded, can you configure SNMPv3 users in the relevant site Web

UI. For more information on configuring SNMP v3 users, see the 5.1 SDC User Guide.

In addition, the NMS Agent SNMP listen port is now 1161 and the OS Daemon port is 161.

Verify that the SNMP configuration is updated accordingly.

To allow port 1161 in the firewall:

Open an ssh session on one of the site master installer servers.

Change the directory by running the following command, where <version> is the version

of your currently installed system:

cd /srv/salt/5<version>/system/

Edit the firewall.sls file and search for the following block of code:

snmp_{{mgmtInterface['ip'~ipv]}}_{{ipv}}:

iptables.insert:

- position: 1

- table: filter

- family: {{ipv}}

- chain: INPUT

- in-interface: {{mgmtInterface['dev']}}

- destination: {{mgmtInterface['ip'~ipv]}}

- proto: udp

- dport: 161:162

- jump: ACCEPT

- match:

- udp

- comment

- comment: "SNMP"

- save: True

- require:

- service: {{iptablesName}}_rules_cleanup





Under this block of code, append the following lines:

snmp_get_walk_{{mgmtInterface['ip'~ipv]}}_{{ipv}}:

iptables.insert:

- position: 1

- table: filter

- family: {{ipv}}

- chain: INPUT

- in-interface: {{mgmtInterface['dev']}}

- destination: {{mgmtInterface['ip'~ipv]}}

- proto: udp

- dport: 1161

- jump: ACCEPT

- match:

- udp

- comment

- comment: "SNMP get and walk"

- save: True

- require:

- service: {{iptablesName}}_rules_cleanup

To apply the changes by running the following command:

salt “*” state.apply system.firewall

To store the system consistency, copy the updated firewall.sls state file to the

second master server.

Validate the change by running the following command:

salt "*" cmd.run "/etc/init.d/iptables status | grep 1161"

and verifying that for servers with nms installed, you see the following output:





ACCEPT udp -- 0.0.0.0/0 10.240.4.117 udp dpt:1161 /* SNMP get and walk

*/

Reinstalling Data Center/Customer Environment RPMs As part of the upgrade process, previously installed Data Center/Customer environment

RPMs were removed. After performing the upgrade, reinstall any relevant Data

Center/Customer environment RPMs. The Data Center/Customer environment specific

RPMs should match the OS version.

Continue Deployment Upgrade If the upgraded site is part of a deployment on multiple SDC sites, managed by an EMS

site, go back to the Upgrading Multiple SDC Sites Managed by an EMS Site to SDC 5.1

from SDC 4.4 section and proceed with the next step.



Upgrading a Standalone SDC Site to SDC 5.1 from SDC 4.4 Prerequisites


3. Upgrading a Standalone SDC Site to SDC 5.1 from SDC 4.4

The SDC 5.1 release can be upgraded to from an existing SDC 4.4 release. This section

describes the upgrade process of a standalone SDC site. If you are upgrading a deployment

of multiple SDC sites managed by an EMS site, follow the procedure described in

Upgrading Multiple SDC Sites Managed by an EMS Site to SDC 5.1 from SDC 4.4.

Note: Since this is a standalone site, this upgrade causes downtime. The expected

downtime is approximately 2 hours.

3.1 Prerequisites

Back Up Site Data It is recommended to create an external backup of the SDC site data folder before beginning

the upgrade.

Back Up the /etc/udev/rules.d/70-persistent-net.rules File The /etc/udev/rules.d/70-persistent-net.rules file contains the site

networking definitions. These definitions may be modified during the upgrade and the

backup file will be needed to support upgrade rollbacks. Create a backup copy of this file

before beginning the upgrade.

Copy the SS7 License and Configuration Files


After the upgrade, the SS7 license must be installed on the upgraded site servers. Before

beginning the upgrade, copy the SS7 license for easy access after the upgrade. In addition,

the SS7 configuration files must be copied to each upgraded SDC site that has an SS7

driver with a CPF.





To copy the SS7 license:

Go to the /opt/DSI/ directory.

Copy the SS7 licenses file - <filename>.lic to a local directory.

To copy the SS7 configuration files:

2. Copy the following files to a local directory:

▪ /opt/DSI/config.txt and /opt/DSI/system.

▪ /opt/traffix/sdc/config/ss7/routing/e212Toe214-v2.csv

▪ /opt/traffix/sdc/config/ss7/routing/mmeToHlr.csv

▪ /opt/traffix/sdc/config/ss7/routing/hss.csv

▪ /opt/traffix/sdc/config/ss7/routing/hlr.csv

Note: If a new version of the SS7 driver is installed during the upgrade, compare the

original configuration files to the new version files. If configuration file sets are not

identical, ensure that the relevant SS7 driver version files are later restored during the

upgrade.

Map the Network Interfaces between Releases The networking definitions previously defined in the SDC 4.4 release may be modified for



This section describes how to create a table of the networking changes. For more

information about how to update the site topology file, see Update the Site Topology File.






Extract the 5.1 release ISO image to a local folder and go to

traffix/repositories/traffix folder.

Locate the salt-srv.rpm package, and extract the data to a local folder.

Go to the scripts folder located at the following path:

..\<customer_defined_local_folder>\salt-

srv<latest_version>.noarch\salt-

srv<latest_version>.noarch.cpio\.\opt\traffix\salt-

srv\<latest_version>\archive.tar\salt\system\scripts

\

Copy this folder to each site server, and run the following commands:

dos2unix showNetMap.sh

chmod +x showNetMap.sh

Run the following script:

./ showNetMap.sh



server:

CURRENT NEW HWADDR











eth2 eth6 ac:16:2d:7d:11:48


eth10 eth8 ac:16:2d:7d:11:49

eth11 eth9 ac:16:2d:7d:11:4d

eth12 eth10 ac:16:2d:7d:11:4a

eth13 eth11 ac:16:2d:7d:11:4e

Generate List of Bonds Using the following command, generate the list of bonds that were used by the SDC 4.4

site. These bonds must be used – without modifications – in the site topology file for SDC

5.1.

To generate the list of bonds used by the SDC 4.4 site:


grep "MII Status\|Duplex" /proc/net/bonding/bond*

Update the Site Topology File The SDC 5.1 release includes a modified site topology file structure. To ensure all statistics

and configuration data are correctly migrated during an upgrade from previous SDC

releases, it is important to understand the current file structure and verify that the existing

definitions for the hostnames, networking configurations (including bonds), and

component names are maintained in the new site topology file.

The high level structural changes between the site topology file structure for SDC release

5.1 and previous releases are detailed in Table 2.





Table 12: High Level Structural Changes


Hosts Vms

Bonds - The bonds are defined for each

virtual machine under a specific

“vm” element in the “vms”

element.

Interfaces - The mapping of each mac to an

interface may change in SDC

5.1. For more information, see

Map the Network Interfaces

between Releases.

Cluster - In SDC 5.1, the SDC

components (applications) are

defined as application instances

per VM, and not as a cluster of

components.

- applications

Networks Networks

Routerules - Not used in SDC 5.1

Routes - The route information is defined

for each virtual machine under a

specific “vm” element in the

“vms” element.

Properties Siteproperties

processes - Not used in SDC 5.1





To create a site topology file to upgrade a site from a previous SDC release to SDC

5.1:

Create a site topology file for an SDC 5.1 site, following the guidelines in the

F5 SDC Guidelines for Creating a Site Topology File.

Under the “networks” element, verify that the previously configured values are defined

under the correct element in the SDC 5.1 site topology file. Table 3 details the relevant

element names and the corresponding name in SDC 5.1.

Table 13: “Networks” Element Level Changes


Id Name

Vlan Vlan

Net net4

ip4sub ip4sub

net6 net6

ip6sub ip6sub

Id Role The valid values for this field

are:

▪ ic

▪ mgmt

▪ sig

▪ custom

Under the “vms” element, verify the following:

There is a “vm” element for each of the “host” elements in the previous topology

file, and the “name” values have not changed.

When defining the vms for an EMS site, verify that the “Issplunkmaster” element

is only set to “true” on the first vm in the site.





The “defaultgateway” element value has not changed, and is defined for each

“vm” element.

Under the “interfaces” element, verify that the previously configured values

are defined under the correct element in the SDC 5.1 site topology file. Table

4 details the relevant element names and the corresponding name in SDC

5.1.

Note: In previous releases, these values were configured under the “vlans”

element.

Table 14: “Interfaces” Element Level Changes


Name network Verify that the value matches the

previously defined value under

the “networks” element.

Id name

Net - This is already defined under the


net6 - This is already defined under the


ip4 ip4



ip6 ip6



Interface Dev






- bondDev Verify that the value matches the



corresponding bond element in

the previous release.

- bondingOpts Verify that the value matches the



corresponding bond_option and

mode elements in the previous

release.

Under the “route” element, verify that the previously configured values are

defined under the correct element in the SDC 5.1 site topology file. Table 5

details the relevant element names and the corresponding name in SDC 5.1.

Note: In previous releases, these values were configured under the “route”

element.

Note: A route can be defined individually for IPv4 or IPv6, but not for both.

Note: Only a route defined as part of an interface that is assigned to a VIP

application will be defined with a source IP.

Table 15: “Route” Element Level Changes


Id name




















24.









24.







Gateway gateway

Fep -






Nic -

src_ip -

target -

sourcenet -

Under each “applicationinstances” element, define the applications that run on

the vm. These applications were previously defined as part of the SDC

cluster. Table 6 details the element names in SDC 5.1.

Note: Application instances must be defined on the same servers (vms/hosts)

as they were in previous SDC releases. In SDC 5.1, all applications run in active-

active mode. The FEP application is the only application that supports active-

standby mode, and this support is provided by defining a VIP application

instance for each FEP instance.

Table 16: “ApplicationInstances” Elements


Type

Name Names are taken from the instance attribute for the resource

element of type app for the application group in the Cluster

element.

Note: If the instance value is empty, the following components

must be named as such:

▪ CPF - cpf1

▪ NMS Agent - nmsagent1

▪ Configuration Manager - traffix_config_mgr-config1

▪ Web UI - webui1

▪ Tripo – tripo1






Other components can be defined with any name.

listenInterfaceName The name of the network interface (s) used by the application.

This must be defined for each application.

Under the “siteProperties” element, verify that the previously configured values are

defined under the correct element in the SDC 5.1 site topology file. Table 7 details the

relevant element names and the corresponding name in SDC 5.1.

Note: In previous releases, these values were configured under the “properties”

element.

Table 17: “siteProperties” Element Level Changes


SITE_ID Name

- sdcVersion

Timezone Timezone

ntpServers ntpServers Define each ntpServer as a sub-

element (“ntpServer”), not as a

string.

traffixFolder traffixFolder

- isManager Define this value as “true” for

an EMS site.

isMultiSiteEnv isMultiSiteEnv Define this value as “true” if the



site.

- splunkAvailable Define this value as “true” if the








site.

EmsServer1

EmsServer2

emsServers Define each emsServer as a sub-

element (“emsServer”), not as a

string.

Note: For SDC site topology files

only.

Update the Installed Data Dictionary Version Release 5.1 includes an updated data dictionary. When upgrading existing systems,

however, this data dictionary is not installed, and the previous data dictionary is kept.

Therefore, to ensure that the updated data dictionary version is installed in the upgraded

site, it must be manually updated to the 4.4 site before the upgrade.

Note: This prerequisite is only relevant for SDC sites installed with release 4.4 CF 18

and earlier.

To manually update the installed data dictionary version:

Download and unzip the SDC release 5.1 package.

Go to the

\opt\traffix\<SDC_release_version>\config\diction

aries folder, and save the superDictionary.xml file locally.

If any manual changes were made to the installed data dictionary, copy those

changes to the saved superDictionary.xml file.





Using the SDC 4.4 Web UI, replace the installed data dictionary with the

updated .xml file, by performing the following steps:

Go to Routing > Data Dictionary. The Data Dictionary screen displays the

currently selected data dictionary.

Click Browse and select the data dictionary file’s location.

In the Protocol field, select the data dictionary’s supported protocol.

Click Submit. The newly selected data dictionary appears in data dictionary list.



Download and install the HP SSA CLI Utility RPM file (hpssacli.rpm) from

the following link:

▪ For RHEL 6:


▪ For RHEL 7:


Validate System Resources An integral part of the upgrade process is the split mirror process. This process designates

one of the HP Arrays as a backup array, ensuring support for upgrade rollbacks. The split

mirror process is supported by a specific HP configuration.

Just before beginning the upgrade, perform the following steps to validate that your HP

configuration is supported:

Note: Perform this prerequisite as close as possible to performing the upgrade.







Validate the RAID Controller type by running the following command:

hpssacli ctrl all show

The supported controller types are:

▪ BL460 gen8, with P220i controller

▪ BL460 gen9 with P244br controller

▪ DL380 gen8 with P420i (+1GB FBWC)

RAID1 / Mirror

RAID 1+0 (Mirror+Stripe, 4 drives)

▪ DL380 gen9 with P440ar (2GB FBWC)

Validate the RAID Controller cache size by running the following command:

hpssacli ctrl all show detail |grep -i “Total Cache Size”

The minimum supported cache size is:

▪ 512MB (Blade)

▪ 1GB FBWC Cache (DL380)

Verify that the RAID Controller has battery backup by running the following command

and getting an OK response:

hpssacli ctrl all show detail |grep -i Battery |grep -i Status

Validate the RAID Controller firmware version by running the following command:

hpssacli ctrl all show detail |grep -i Firmware

The minimum supported firmware version is:

▪ RAID Adapter Firmware Version:

▪ 5.42 and later



Upgrading a Standalone SDC Site to SDC 5.1 from SDC 4.4 Performing the Site Upgrade


▪ 4.68

Validate TC Timer Parameter The TC Timer parameter (Topology > Specific Site Settings > Site> SDC Components

or Topology > Remote Peers) in 4.4 cannot be defined as more than 30,000 milliseconds.

If it is, then the Reestablish Connection Time parameter in 5.1 cannot be validated and any

other edits to the SDC Component Properties cannot be saved.

3.2 Performing the Site Upgrade

Stop the Corosync Cluster Stop the Corosync cluster on the site servers.

Note: The site servers will no longer process traffic until the upgrade is successfully

completed.

To stop the Corosync cluster:

Run the following command on each site server:

Note: Run this command on the site servers actively running the SDC components

before the site servers running the SDC component in standby mode.

crm node standby

Back Up Collected KPI Data To create a file of all collected KPI data:

Run the following commands on the servers running the active NMS Agent(s)

in the SDC site:





cd /opt/traffix/sdc/data/backup/statistics/

tar cvfz nmsagent.tar.gz nmsagent

Perform a Split Mirror During this step, a backup array is created to enable upgrade rollbacks.

To create a backup array:

Run the following command on each site server to switch the servers to

standby mode.

crm node standby

Perform the Split Mirror procedure to designate one of the HP Arrays as a backup array:

Run the following command to verify that the logical drive is in an “OK” state:


Run the following command to create a backup array:

hpssacli controller slot=0 array A splitmirror action=splitwithbackup

Enter “y” to confirm that you want to continue after being prompted with the

warning message.

Run the following command to verify that there are two arrays, each with a logical

drive, and that backup logical drive was created.


Note: To ensure successful rollbacks, be sure to clearly mark which array and

logical disk are the backup.

Run the following command to create a visual marker for the mirrored array:

hpssacli controller slot=0 <active_array_ID> modify led=on





Set up the Site Machines In this phase, the site machines are rebooted and the master and minion servers are created.

Note: Always set up the master Installer servers and upload the site topology file to

them before setting up the other site machines.

You need to install the operating system on each site machine. The operating system is

installed from the ISO image.

Note: You can load the ISO image using the ILO Integrated Remote Console, any other

available console or directly from your system.

Install the Operating System In order to install the operating system, you need to load the ISO image.

Note: The ISO image must be loaded for each server in the site.


Note: For an EMS deployment, the following steps can be run in parallel on each EMS

site.

Select the ISO image from where it is saved on your computer.

Start the installed site machine from the ISO image.

The Welcome To F5 Traffix SDC Install Menu is displayed.

Under the Welcome To F5 Traffix SDC Install Menu, select Install Traffix F5 EL

from cdrom for bare metal.

The GRUB boot loader page displays.





Figure 2: GRUB Boot Loader Page

Define Master and Minion Servers The GRUB boot parameters define a server's role as either a master Installer server or as a

minion server that will host the SDC components. Configuring the parameters is done from

the GRUB boot loader page. There are mandatory parameters and optional parameters that

are only required if relevant for the deployment.

To configure the GRUB boot parameters:

In the prompt line, after F5-TRAFFIX_SDC:traffix/kickstart/kickstart.cfg,

press the TAB key to enable editing and add the parameters, as follows:





Table 18: Mandatory Parameters


Server master/minion

Hostname the server's hostname

Note: The hostname must be identical (case sensitive) to the value

defined under the name attribute for the vm element in the Site

Topology file

master0 The IP address on the management network that the first vInstaller

uses.

master1 The IP address on the management network that the second

vInstaller uses.

Ip The IP address is from the management network interface for

minion and master Installer servers.

Netmask netmask for the IP address defined above (CIDR is not supported

on IPv4)

Device The ethernet interface used by the IP address defined above

Table 19: Optional Parameters


Vlan vlan number for interface(if vlan defined)

Gw default gateway (need to be mandatory if server = master)

debug debug=yes enable salt log with debug

Dns DNS

The following is an example of inputted GRUB parameters:





Click ENTER when done entering the parameters.

Repeat these steps for each site machine.

Each site server is now installed with an Operating System and has a defined role

(master or minion). You can verify which servers in a site are defined as a master

or minion with the siteStatus API Request.

Modify the GRUB Boot Parameters The GRUB boot parameters are saved in the params file. If you want to change any of the

parameters or add a parameter, you can do so by editing the params file.

Note: You can only edit the parameters at this stage in the installation process, prior to

uploading the Site Topology file. After editing any GRUB boot parameters in params File,

you must run an installation script.

You can reconfigure a master server as a minion server, but you cannot reconfigure a

minion server as a master server.

To edit the GRUB boot parameters:

Go to the directory where the params file is saved:

# cd /var/tmp/salt-install/

Execute the following command to edit the relevant parameters:

vi params

Note: If when entering the GRUB parameters from the GRUB boot screen, you

clicked ENTER, but you still want to edit the parameters, you need to use the example

params file to edit by executing the following command:

cp params.example params





Add/Edit a parameter according to the list of mandatory or optional parameters (see

Table 8 and Table 9).

Execute the following command to run the installation script:

./salt-install.sh

Each installed site server is now updated with the edited parameters. You can

verify which servers in a site are defined as a master or minion with the siteStatus

API Request.

Note: If you changed the IP address of the master Installer or any of the GRUB

parameters in the params file, then you need to run the installation script on all of the

minion servers. If the master Installer IP address has not been changed, then you need

to restart the Salt minion service.

Restart the EMS NMS Server It is recommended that you restart the EMS NMS server after the upgrade of all SDC sites

has been completed. If not, the EMS Web UI may not accurately reflect the correct status

(Up, Down) of the SDC site components.



Run the following command on all site servers:

yum install hpssacli

Load the Backed Up Site Data Once the site servers are successfully created, the backed up site data is loaded from the

logical drive previously defined as the backup logical drive.





Migrate the backed up site data saved in the Config Manager:

Enable access to the backup disk created with the Split Mirror mechanism:

i. Run the following command on the active array on each site machine:

hpssacli controller slot=0 <active_array_ID> splitmirror

action=activatebackup

Warning: Potential loss of site data!

Once this step has been performed, the logical drive is no longer defined as a backup

drive, and its data is no longer protected from being overwritten. Any ISO installation

performed after this point will result in the data on both logical drives being

overwritten. To ensure that the data on the backed up logical drive will not be lost,

follow the rollback procedure to return the site servers to their pre-upgrade state

(running SDC 4.4) before starting the upgrade from the beginning.

Run the following command to ensure that the logical drive that is used is the logical

drive that is being upgraded and is running with SDC release 5.1:

hpssacli controller slot=1 ld <ID of the active Logical Drive> modify

bootvolume=primary

Run the following commands on both site machines to identify the activated disk

that was previously backed up:

grep device /etc/lvm/backup/vg1 -B1 | grep id

pvs –o pv_name,vg_name,pv_uuid

The output of the first command includes the disk that the system recognizes,

while the output of the second command includes a list of all the disks that are

installed in the system.





Identify the disk in the output of the second command that did not appear in the list

of recognized disks (in the output of the first command), and run the following

command using the disk name of the disk that did not appear in the list of

recognized disks:

vgimportclone --basevgname vg1_clone /dev/sd<name of disk>

Run the following commands to mount and activate the volume that you want to

restore data from:

lvchange -a y vg1_clone/lv_opt

mount -t ext4 -o ro /dev/mapper/vg1_clone-lv_opt /mnt

Note: If the site must be re-installed after this stage, perform a rollback. For more


Verify that the /data folder (or its subdirectories) is available for data migration:

i. Locate the /data folder (or its subdirectories) in the site topology file and

verify that it has been defined as a “mountPoint” value, meaning, to be

used as a volume. Once this has been verified:

ii. Copy the updated site topology file to each site machine.

iii. On each site server, run the emsMigration.sh script using the –m argument,

followed by the full path to the topology file:

/srv/salt/<current version>/system/scripts/emsMigration.sh -m

<path_to_topology_file.xml>

Copy the previous backup folder from the backup disk to the verified data folder:

i. On each site server, run the following command:

cd /mnt/traffix/<sdc_version>/data





cp –rp backup /data/

Note: After loading the backed up site data, ensure that in the case of a reboot the

active array will be rebooted as the primary array by running the following command:

hpssacli controller slot=<relevant slot ID> ld

<active_array_logical_drive_number> modify bootvolume=primary

hpssacli controller slot==<relevant slot ID> ld

<backup_array_logical_drive_number> modify bootvolume=secondary

hpssacli controller slot==<relevant slot ID> ld 3 modify bootvolume=none

Uploading the Site Topology File In this phase, the site topology file is uploaded to the master Installer servers using an API

request. Before executing the API request to upload the Site Topology file, you need to

validate the Site Topology File and then upload it.

Note: Upload the Site Topology file only after the master Installer servers are up and

running.

To validate the Site Topology File:

Run the following command from cd /srv/traffix:

#python pillar/traffix_validate.py /tmp/ topology.xml

The following is an example of a successful validation:

validate_topology_string: succeeded to validate xml file topology=<?xml

version="1.0" ?>

The following is an example of a failed validation:





validate_topology_string: failed to validate xml file topology=<?xml

version="1.0" ?>

To upload the Site Topology file to the Installer Machines:

Upload the Site Topology file to a master Installer server.

the Installer REST interface by sending the following request to the master

Installer to generate an authentication token:




eauth='pam'

Note: For all API requests, you need to use the minus sign, for example "-d" and

not the N-dash "-". If you copy–paste the API request, you may have to type in the "-

d" again with the minus sign to avoid syntax conversion errors.

The success return code for this request is ‘200’.

After validating the Site Topology file and generating an authentication token, run the

following API command to upload the topology file:

curl -ksi https://<master_IP_address>:8000 -H "Accept: application/x-

yaml" -H "X-Auth-Token: <Token>" -d client="runner" -d

fun="traffix.uploadTopology" -X POST -d @topology.xml

The response indicates if the Site Topology file has been successfully uploaded. The

following is an example of the API request with a successful response:

# curl -ksi https://localhost:8000 -H "Accept: application/x-yaml" -H

"X-Auth-Token:aa6790aa67ae5ce87715b66bf5bd58fc3ea4bdb5" -d





client="runner" -d fun="traffix.uploadTopology" -X POST -d

@topology.xml

HTTP/1.1 100 Continue

HTTP/1.1 200 OK

Content-Length: 63

Access-Control-Expose-Headers: GET, POST

Access-Control-Allow-Credentials: true

Vary: Accept-Encoding

Server: CherryPy/3.2.2

Allow: GET, HEAD, POST

Cache-Control: private

Date: Tue, 15 Dec 2015 13:27:04 GMT

Access-Control-Allow-Origin: *

Content-Type: application/x-yaml

Set-Cookie: session_id=6db032df27701c9dfa9dbbf9d8a8a42c682dbb68;

expires=Tue, 15 Dec 2015 23:27:04 GMT; Path=/

return:

- - 0

- topology uploaded to the server successfully

The Site Topology parameters are now saved in the Cassandra database in both of the

master Installers. Once the Site Topology file has been uploaded successfully to the

master Installer servers, set up the remaining site machines by following the Setting

up the Site Machines procedure.

Note: Once the Site Topology file has been uploaded successfully, the only way

to modify the site configurations (in the params file or in the Site Topology file) is to

perform a new installation by reinstalling the ISO.





Installing the SDC Components The minion servers communicate with the master Installer servers and the master Installer

servers then reply to the minion servers, based on the Site Topology definitions, to know

where to install the different SDC components (FEP, CPF, etc.).

Verifying the SDC Installation Status This verification is done by invoking the “appStatus” REST API. Applying this REST

APIs queries the master Installer about the status of the SDC applications running on a

specific server or on all the servers. This API is based on a standard Salt API interface and

the body of the REST API message contains CLI Salt functions.

The site is only successfully installed once the “12000” result code is returned for the

appStatus API request.

Generating an Authentication Token Before any REST API request can be sent, you must have a valid authentication token. You

need to send a request to the master Installer to generate an authentication token.


To generate an authentication token:

Send the following API request to the Master Installer that is identified by the

<master_IP_address> parameter:



eauth='pam'

Note: For all API requests, you need to use the minus sign, for example "-d" and not

the N-dash "-". If you copy–paste the API request, you may have to type in the "-d" again

with the minus sign to avoid syntax conversion errors.





Authentication Request Status Codes The following are the possible return codes for the authentication API request:

Return Code Description

200 Success

401 authentication required

406 requested Content-Type not available

Application Status per Server This API request checks the status of a specific server. The response includes the relevant

status codes for successfully installed applications. In addition, as with all other API

requests, there are related command execution codes.

appStatus API Request curl -ksi https://<master_IP_address>:8000 -H "Accept: application/x-

yaml" -H "X-Auth-Token:<Token>" -d client="runner" -d

fun="traffix.appStatus" -d tgt="*" -d apps=True (optional for apps

list)

Command Execution Codes for appStatus API Request Table 20: appStatus Command Error Codes


-50 Failed to validate site topology file - check site topology file

-51 Installation not started yet

-52 Could not get information from DB

Return Codes for appStatus API Request Table 21: appStatus Return Codes


14002 Pending Machine Start



Upgrading a Standalone SDC Site to SDC 5.1 from SDC 4.4 Post Upgrade Tasks



14003 Pending SDC Installation

14004 Pending SDC Start

14006 Pending SDC Stop

15002 Fail VM Start

15003 Fail To Install SDC

15004 Fail To Start SDC

15006 Failed To Stop SDC

13000 Suspended

12000 Successfully installed

3.3 Post Upgrade Tasks

Restore Backed Up Site Data In this phase, restore the data from the created folder:

Copy the created 4.4 backup SDC KPI data .tar file to the /data folder on

one of the site servers by running the following command, making sure that

the “site_id” value is the same as the value defined for the SDC site name in

the site topology file::

cp –rp /mnt/traffix/<site_id>/data/backup/statistics/nmsagent.tar.gz /data/

Run the following script on the same servers:

/opt/traffix/scripts/kpiMigration.sh -s <site_id>

Verify that the following files were successfully created:


Stop and start the NMS Agent in the SDC site by running the following command on

the server that the file was copied to:





monit stop <server_name><nmsagent_name>

monit start <server_name><nmsagent_name>

Verify that the restore is completed by checking in the


file for the following message:

upgrade_status=upgrade_done

Delete the .tar file(s) from the /data folder.

Disable Session Data Replication by Tripo The session data saved in the Tripo (session repository) is replicated between session

repositories on geo-redundant SDC sites. Since this is a standalone site, this default

replication must be disabled.

To disable Tripo-based session data replication:

Run the following commands on each server running Tripo Resource:

su - traffix


. DefEnv Tripo

UI_Config

set SiteReplication false

CTRL–C to exit from UI_config

UI_Config -w ConfigParams.cfg

Verify that Tripo-based session data replication was disabled by running the following commands:

Run UI_config

==>dump





Review the ConfigParams.cfg file (located in the /home/traffix/Tripo/cfg/ folder)

and verify that SiteReplication = false

Perform a Site Health Check Verify that the site is running as expected by performing a health check:


/srv/traffix/upgrade/upgrade-cli

Log in to the upgrade CLI, using traffix as your username and password.

Note: If you have changed your Web UI password since the upgrade, use the new

password to log in to the upgrade CLI.

Run the following command in the upgrade CLI:

healthcheck

If site components are up and running as expected, the EMS site has been

successfully upgraded. If not, perform an upgrade rollback. For more


Re-Mirror the Drives After verifying that the site is up and running as expected, re-mirror the drives by

performing the following procedure, based on the site that was upgraded:

▪ After upgrading an EMS site, re-mirror the drives by performing the Re-Mirroring

the Drives on an EMS Site procedure.

▪ After upgrading an SDC site, re-mirror the drives by performing the Re-Mirroring the

Drives on an SDC Site procedure.





Re-Mirroring the Drives on an EMS Site

Warning: Performing this procedure removes the option to roll back the upgrade on this

site.

To re-mirror the drives on an EMS site:



Enter HP SSA in the remote console.

Reboot the server.

At the prompt, enter HP Storage Controller Administrator.

Press F5 (for Gen8) or F10 (for Gen9) to start the HP Storage Controller

Administrator.


steps:

Under Array Controller(s) select the backup array

From Actions list - click Configure.

Under Controller Devices, select Arrays.

Under Arrays, select the active array.

Under Actions select Re-Mirror Array.

Re-Mirroring the Drives on an SDC Site

Warning: Performing this procedure removes the option to roll back the upgrade on this

site.





To re-mirror the drives on an SDC site:

Run the following command on the active array on each SDC machine:

hpssacli controller slot=0 <active_array_ID> splitmirror action=remirror

Run the following command to check the progress of the re-mirroring:


The output indicates the percentage that has been re-mirrored, for example:

Smart Array P220i in Slot 0 (Embedded)

array A

logicaldrive 1 (279.4 GB, RAID 1, Recovering, 23% complete)

The re-mirror is complete once the output shows the logical drive status as “OK”.

Install the SS7 License


Each server running SS7 must have a license installed.

To install the SS7 license, perform the following step on every server that will run

SS7:

Copy the saved SS7 license file to /opt/DSI

Stop and start the CPF component by running the following:







Add Licenses to New FEP IP Addresses Each FEP IP address must have a license. During the upgrade, additional IP addresses may

have been added to the FEP instances. If additional IP addresses were added, these IP

addresses much each have their own license. For more information about obtaining the

license, contact F5 Support and refer to the F5 SDC User Guide on how to add a new

license key.

Modify Existing SNMP Configurations The system alarms and KPIs were modified in release 5.1. Therefore, the previous SNMP

configurations and related scripts must be adapted according to the 5.1 MIB file. Verify

that the SNMP configurations have been updated according to the MIB file included in

release 5.1.

In addition, the NMS Agent SNMP listen port is now 1161 instead of 161. Verify that the

SNMP configuration is updated accordingly. As of release 5.1 CF 9, all upgraded sites,

will be enabled with SNMPv3 functionality. For more information about SNMPv3, refer

to the 5.1 User Guide.

Note: Following a completed upgrade, the SNMP V2 Trap Forwarding profile will

include the default settings. You need to manually configure these settings (community

string) per SDC site after the upgrade is completed. Refer to the F5 SDC User Guide for

more information on configuring the SNMP V2 Trap Forwarding profile.

Reinstalling Data Center/Customer Environment RPMs As part of the upgrade process, previously installed Data Center/Customer environment

RPMs were removed. After performing the upgrade, reinstall any relevant Data

Center/Customer environment RPMs. The Data Center/Customer environment specific

RPMs should match the OS version.





Enable Wifi Offload

Note: This step is only relevant for SDC sites running Wifi Offload.

Enable Wifi Offload by performing the following steps on all site servers:

Run the following commands:

cd /opt/traffix/sdc5.1_1127/config/sysconfig

vi traffix_cpf


Remove #

WIFI_OFFLOAD_RADIUS=com.traffix.openblox.eap.RadiusAuthenticati

onStandardStackFactory

WIFI_OFFLOAD_SS7=com.traffix.openblox.eap.SS7AuthenticationStand

ardStackFactory

WIFI_OFFLOAD_DIAMETER=com.traffix.openblox.eap.DiameterAuthe

nticationStandardStackFactory

Run the following commands to stop and start the CPF component:





Rolling Upgrade to a New Release Version Prerequisites


4. Rolling Upgrade to a New Release Version You can perform a rolling upgrade from a 5.1 release to a new 5.1 release version. The

rolling upgrade is performed using Salt commands from the CLI application. It includes


Uploading the new ISO image to a Master Installer (Component Upgrade)

Upgrading the Master Installer

Upgrading the OAM database

Upgrading the SDC components

Upgrading the operating system

Throughout the upgrade, you can monitor the process and check to see which SDC

components are up and running. In the event that certain components are not successfully

upgraded, you can choose to either resume the upgrade process or perform a rollback.

While the length of an upgrade process varies with the size of the deployment, for a

deployment with four servers (and ten SDC components), the upgrade process takes

approximately 40 minutes, with minimal downtime. After completing the upgrade of all

the components (applications), you need to upgrade the operating system.

4.1 Prerequisites This section describes the prerequisites of a rolling upgrade.

CLI Application This upgrade process uses the upgrade CLI Application. To proceed with the upgrade

process, you must be logged in to the CLI Application.

To access the upgrade CLI Application:


cd /srv/traffix/upgrade/





./upgrade-cli

Log in with your SDC Web UI username and password.

Figure 3: Upgrade CLI Application Login

New ISO Image The ISO image contains the operating system and it is packaged and provided as bootable

media by F5. Verify that you have saved the ISO image in a location that you can later

point to to load it.

Pre-Upgrade Health Check It is recommended that you check the status (disconnected/connected) of the SDC

components prior to performing the upgrade. In this way, you will have a baseline to

compare which SDC components are connected during and after the upgrade. The health

check is performed with the CLI command: healthcheck

The following is an example of a healthcheck





Updating the Salt Files If you are upgrading from a 5.1 CF10 or lower release to a 5.1 CF 11 or higher release than

you must do the following procedure to avoid a Salt process collision while upgrading.

This procedure enables you to run multiple Salt process simultaneously without any

process collision.

Note: This procedure is required for both EMS and SDC site deployments. This

procedure needs to be performed on both master Installer servers.

To update the Salt files:

Backup the /srv folder on each of the master Installer servers.

Create a temporary folder:





mkdir /tmp/patch

Upload the salt-srv package from the ISO of the new CF to the /tmp/patch folder.

Copy the new CF ISO to the server to the relevant <path>, for example: /root/

Mount the new CF ISO:

mkdir -p /mnt/tmp_mount

mount -o loop /<path>/iso-<New CF ISO Version>.iso /mnt/tmp_mount/

Copy the new Salt version from the mounted ISO:

cp /mnt/tmp_mount/traffix/repositories/traffix/salt-srv<version>.rpm

/tmp/patch/

Unmount the ISO:

umount /mnt/tmp_mount

Extract the rpm contents:

Go to cd /tmp/patch/


rpm2cpio salt-srv< New CF ISO Version >.noarch.rpm | cpio -idmv

Go to the following folder and extract the file:

cd /tmp/patch/opt/traffix/salt-srv/<New CF ISO Version>/

tar xvf archive.tar

Remove the old Salt files saved in the /srv/traffix folders:

rm -rf db_utils/ external_api pillar runners statusApi traffix_models

upgrade utils





Figure 4: Example of Removed Salt Files

Copy the new Salt files from /tmp/patch/opt/traffix/salt-srv/<RPM VERSION>/traffix

into the /srv/traffix folders:

Go to cd /tmp/patch/opt/traffix/salt-srv/<New CF ISO Version>/traffix/


cp -rf db_utils external_api pillar runners statusApi traffix_models upgrade

utils /srv/traffix/

Figure 5: Example of Copied New Salt Files

Once the new Salt files are saved in /srv/traffix, you need to change the file status to be

executable:

Go to cd /srv/traffix/upgrade

Run the following command: chmod +x upgrade-cli





Backup the original Salt-minion files before copying the new files which will overwrite

the original ones.

Locate the original files in /usr/bin/ and etc/init.d/ and back these files up to a

temporary folder:

cp salt-minion /var/tmp/

Copy and enable the following new Salt-minion files from the new ISO under

/srv/traffix/pillar/ to /usr/bin/ and /etc/init.d/, respectively:

▪ usr_bin_salt_minion

▪ etc_initd_salt_minion

Copy the new Salt-minion files, with the following commands:

cd /srv/traffix/pillar

cp usr_bin_salt_minion /usr/bin/salt-minion

cp etc_initd_salt_minion /etc/init.d/salt-minion

Enable the file for execution:

chmod +x /usr/bin/salt-minion

chmod +x /etc/init.d/salt-minion

Note: Repeat this step on all the servers in the deployment.

Once the new Salt files have been copied on all the servers, run the following

command:

monit restart salt-minion

Note: Repeat this step on all the servers in the deployment.



Rolling Upgrade to a New Release Version Uploading the New ISO image for the Component Upgrade


Setting the External Port Range In Release 5.1 CF 10, the external port range was redefined to 50000-59999 from the Linux

default of 50000-64000. This change was done to prevent port collision between SDC

components and other components, such as the NMS Agent and configuration manager

that have set port numbers 61617 and 61616, respectively.

To ensure backward compatibility for port connections, you need to redefine the external

port range for SDC components.

To set the external port range:

In each of the Master Installers, go to the following file:

update /srv/salt/5.1_8-7/system/sysctl/sysctl.conf file

Change the following:

net.ipv4.ip_local_port_range = 50000 64000

To:

net.ipv4.ip_local_port_range = 50000 59999

Note: If using IPv6 protocol, make the change to the net.ipv4.ip_local_port_range

as the IPv6 protocol relies on the IPv4 parameters.

From one of the Master Installers, run the following command:

salt '*' state.highstate

3. Reboot, one at a time, to those servers that host a FEP component.

4.2 Uploading the New ISO image for the Component Upgrade You first need to upload the new ISO image to one of the Master Installers. Once the new

ISO image is mounted, it needs to be copied to the /opt/repo/traffix folder.



Rolling Upgrade to a New Release Version Upgrading the Master Installer



Perform the required pre-installation configurations needed for the console that

is being used. The following steps assume that the ILO Integrated Remote

Console is being used:

Configure the ILO IP address.

Connect to the ILO Integrated Remote Console and select Virtual Drives.

Select Image file CD -ROM/ DVD to set with ISO file.

Mount the ISO image from where it is saved on your computer.

Run the following command to copy the new ISO image to the /opt/repo/traffix folder:

copyFromIso

Note: If the copyFromIso procedure is not successfully carried out, the following

message is displayed to notify the user that the relevant RPMs were not copied to the

second Master Installer:

"Could not copy repo to second master"

4.3 Upgrading the Master Installer During this phase, the Master Installers are upgraded with the operating system included

in the new ISO image.

To upgrade both master installers:


Saltupgrade



Rolling Upgrade to a New Release Version Upgrading the OAM Database


Note: The saltUpgrade command checks for any manually applied changes in the salt

state files. If there are any, the following message appears in the upgrade CLI interface and

in the logs: “Salt state files have been manually modified.” The saltupgrade automatically

generates a patch file located in /srv/salt/salt.patch before exiting from the CLI

Application. You need to copy any newly created set of state patch files to the new state

files in /srv/salt/<51.-new rpm salt srv version>.

Once the files have been copied, and only in the case the saltupgrade was interrupted

because of changes in the state files, continue with upgrading the Master Installers and then

run the saltEngineUpgrade command.

After running the Saltupgrade or the saltEngineUpgrade command, verify that the

Master Installer components upgrade status is “started” with the following command:

SELECT * FROM statusflow.appflow

After running this command, you will be logged out of the CLI Application.

4.4 Upgrading the OAM Database During this phase, the OAM database is upgraded with the data included in the new release

version.

Note: You must log in again (with the following command: ./upgrade-cli) to the CLI

Application before proceeding with this step.

To upgrade the OAM database:


upgradeOamDB



Rolling Upgrade to a New Release Version Upgrading the Tomcat Version


4.5 Upgrading the Tomcat Version During this phase, the Tomcat version is upgraded with the version included in the new

release version.

Note: You must be logged in to the CLI Application before proceeding with this step.

To upgrade the Tomcat version:


tomcatUpgrade

4.6 Upgrading the Splunk Version During this phase, the Splunk version is upgraded with the version included in the new

release version.

Note: You must be logged in to the CLI Application before proceeding with this step.

To upgrade the Splunk version:


splunkUpgrade

Verify that Splunk has been upgraded with the healthcheck command. If the

healthcheck is not valid, run the splunkUpgrade force command.

4.7 Upgrading the SDC Components During this phase, each SDC component is upgraded. The system knows inherently to

upgrade each SDC component by component layer, starting with Tripo. Once the CLI

command is executed, an internal process is carried out with the following phases:



Rolling Upgrade to a New Release Version Restarting the API Flows Post-Upgrade


preUpgrade, stop, install, update Links, postUpgrade (includes an nmsHealthCheck, to see

if Upgrade was successful).

To upgrade the SDC components:


upgrade

Note: Only components that are connected, can be upgraded.

Resuming the SDC Component Upgrade In the event that the SDC component upgrade stops in the middle of the process, you will

see a prompt that the "Upgrade is not successful." It is recommended that you investigate

the root cause of the failure, correct it and then resume the upgrade. .

To restart the SDC Upgrade process:


resume

4.8 Restarting the API Flows Post-Upgrade Upon completing the upgrade and checking that all SDC components are up and running,

you need to unlock the API flows that were stopped during the upgrade. This essentially

resets the API request flows to an idle state so the system is fully ready to run all API

requests.

To restart the API flows:


finalizeUpgrade



Rolling Upgrade to a New Release Version Upgrading the Operating System


Note: Initially, after running this command, you may get a response that the "lock is

busy" and that the API request "flow lock not idle". In this case, run the command again

after thirty seconds, and the following response is displayed:

Unlocking flow

enabling highstate on all hosts

lock is idle

4.9 Upgrading the Operating System After the SDC components have been upgraded, you must upgrade the operating system.

Note: Always perform the OS upgrade first on the machines that have Tripo on them.

Prerequisites for Upgrading the Operating System This section describes the prerequisites for upgrading the operating system. You might

have already performed these prerequisites when upgrading the SDC components.

New ISO Image The new ISO image with the new operating system should already be uploaded and

accessible as part of the component upgrade.

Copy Data Center/ Customer Environment RPMs

The upgrade process includes an OS installation. This upgrade removes any installed Data

Center/Customer environment specific RPMs. Copy the RPMs that you want to reinstall

after the upgrade.

Split Mirror for Rollback Perform HP Split Mirror with Salt (this might have been done as part of the APP upgrade).

For more information, see Performing a Split Mirror.





Pre-OS Upgrade Health Check Prior to upgrading the operating system (OS) with the new RPMs, you should check the

status of the SDC components. It is recommended that you save this healthcheck for

comparison purposes post-upgrade to check if any components that were previously up are

down post-upgrade.

To check the status of the components post-component upgrade:

If you are not logged in to the CLI Upgrade application, do so. For more

information, refer to CLI Application.


healthcheck

Uploading the New ISO Image In this step, you will load the ISO and save the RPM packages in repo folders.

Note: At this phase, you must be logged out of the CLI application to continue with

the upgrade.

To upload the ISO:


/opt/traffix/scripts/osRollingUpgrade.sh –l

Note: Only the latest version of each loaded RPM package is saved in the repo

folder. If you need to remove any packages that were uploaded by mistake, do the

following:

Log in to each master Installer server.

Delete the relevant file from /opt/repo/updates





Run the command: createrepo /opt/repo/updates

Upgrading the Operating System The upgrade must be performed remotely for each server (hostname) in the site. Remote

actions are done using the Salt cmd.run module. After upgrading the RPM package, you

need to restart the SDC components on the server. Prior to restarting, it is recommended

that you check which components are up and running on the server.

To upgrade the RPM package:


salt 'hostname*' cmd.run "/opt/traffix/scripts/osRollingUpgrade.sh -u"

Note: If you need to see a list of the host names, run the following command:

grep "# host" /etc/hosts

To restart the server (host):

Check which SDC components are up and running on the server with the

upgraded OS:

salt '<hostname>*' monit.summary

Stop the SDC component services:

salt '<hostname>*' monit.stop <SDC instance name>

Note: The OAM database (cassandra) must be the last service to be stopped.

Restart with the following command:

salt '<hostname>*' cmd.run reboot

Note: Wait for the server to finish rebooting.



Rolling Upgrade to a New Release Version Monitoring the Rolling Upgrade Process


Start all SDC component services:

salt '<hostname>*' monit.start all

Repeat the above steps for all the site servers.

4.10 Monitoring the Rolling Upgrade Process Throughout the rolling upgrade process, there are a few ways to monitor the process. In

the event that you want to roll back the upgrade process, you can do so. For more


Validating the SDC Processes Post-OS upgrade, you should check to see that all the processes are up and running. For

examples. You should see the following processes:

▪ vInstaller (master)

▪ cm (config manager)

▪ nms (nms agent)

▪ oamDB (cassandra)

▪ salt-api

▪ salt-minion

▪ cpf

▪ fep

▪ Tripo

▪ Splunk

To check the processes:


salt '*' monit.summary



Rolling Upgrade to a New Release Version Monitoring the Rolling Upgrade Process


Monitoring the Status of SDC Components The CLI Upgrade application command healthcheck checks the connectivity status of each

SDC component. This command can be executed throughout the upgrade process. See Pre-

Upgrade Health Check for more information and an example.

Monitoring SDC Release Versions You can check which SDC release versions are available to support your deployment.

Upon completing the upgrade, the previous version and the new version should be

displayed.

To view the SDC release versions:


ls -l /opt/repo/traffix/

Figure 6: Example of Monitoring SDC Release Versions

Monitoring the Upgrade Logs You can check the upgrade logs for general troubleshooting. The upgrade logs are located

in the /var/log/upgrade.log folder.

Monitoring Salt Packages You can check which salt-srv packages are being used.

To view which Salt packages are being used:


- yum search salt-srv



Performing an Upgrade Rollback to 4.4 Rolling Back an SDC Site Upgrade to Release 4.4


5. Performing an Upgrade Rollback to 4.4 The upgrade process includes the option to roll back an upgrade that has encountered

errors.

▪ To roll back an upgrade of an SDC site managed by an EMS site, follow the Rolling

Back an SDC Site Upgrade procedure.

▪ To roll back an upgrade of an EMS site before activating the backup array, following

the Rolling Back an EMS Site Upgrade to Release 4.4 before Activating the Backup

Array procedure.

▪ To roll back an upgrade of an EMS site after activating the backup array, follow the

Rolling Back an EMS Site Upgrade to Release 4.4 after Activating the Backup Array

procedure.

5.1 Rolling Back an SDC Site Upgrade to Release 4.4 This procedure is used to roll back an upgrade of an SDC site that is managed by an EMS

site. This procedure uses the backup array as the master copy.

Note: After rolling back an SDC site that is managed by an EMS site, the EMS global

configuration parameters will be distributed to the local SDC site. Any local configuration

changes made on the local SDC site after the upgrade and before the rollback will be

deleted from both the local SDC site and from the EMS site that manages it.

To perform the upgrade rollback:

Stop all traffic on the SDC site that you want to rollback.

Stop the applications running on this SDC site and the EMS site by running the

following command on the relevant servers:

monit stop all





Run the following commands on the EMS site server to delete the SDC site

data:

cd /data/backup

mv <site name> /var/tmp/

From the EMS mount folder, copy the 4.4 site data to the EMS /data/backup/

folder:

cp -rp /mnt/traffix/<sdc-version>/data/backup/<Site Name> data/backup/

Start the EMS site:

monit start all

While the SDC site is still down, on the EMS site, run the following command to

generate the Cassandra class status:

/opt/cassandra/bin/nodetool status

The following is an example of the Cassandra class status for EMS Datacenter:

Status=Up/Down

State=Normal/Leaving/Joining/Moving

-- Address Host ID

Rack

UN 10.240.13.67 d126cb8a-b3e0-4a8c-a4aa-453d88316f1e RAC1

UN 10.240.13.68 33eeb22a-edf0-4a52-89c7-54367b3f55a2 RAC1:

The following is an example of the Cassandra class status for SDC site

Datacenter:

Status=Up/Down

State=Normal/Leaving/Joining/Moving

-- Address Host ID

Rack

UN 10.240.13.69 7e5a6a30-2c2e-4a7c-bfb1-1b8508125916 RAC1





UN 10.240.13.70 b9c50a48-c046-4383-a759-5ec1ae3f6ecb RAC1

From the Cassandra class status output, remove the SDC Site’s host IDs, by selecting

them and then running the following command:

/opt/cassandra/bin/nodetool removenode <host ID>

For example, based on the generated SDC site datacenter output, the commands

would be the following:

/opt/cassandra/bin/nodetool removenode 7e5a6a30-2c2e-4a7c-bfb1-

1b8508125916

/opt/cassandra/bin/nodetool removenode b9c50a48-c046-4383-a759-

5ec1ae3f6ecb

Clear the Cassandra data base keyspace of the rollback SDC site, by running the

following script:

python /opt/traffix/scripts/clearSdcSiteFromEms.py <SDC rollback Site

name>

On the SDC site, re-mirror the two arrays using the backup array:

Access the HP Storage Controller Administrator BIOS Interface by performing the

following steps:

i. Enter “HP SSA” in the remote console.

ii. Reboot the SDC site server.

iii. At the prompt, enter “HP Storage Controller Administrator”.

iv. Press F5 (for Gen8) or F10 (for Gen9) to start the “HP Storage Controller

Administrator”.


steps on the SDC site servers:

i. Under "Array Controller(s)" select the active array.



Performing an Upgrade Rollback to 4.4 Rolling Back an EMS Site Upgrade to Release 4.4 before Activating the Backup Array


ii. From Actions list - Click "Configure"

iii. Under "Controller Devices", select "Arrays"

iv. Under "Arrays", select the active array.

v. Under "Actions" select "Manage Split Mirror Backup".

Define the rollback procedure:

i. Select the following option: "Re-mirror the array and rollback the contents

of the backup array. Discard existing data", to roll back to the previous

OS image, make it the master and sync the upgraded drive to it.

ii. Confirm the selection.

Click “X” at the top right corner of the SSA Menu screen, and confirm the exit.

Click the Power Icon at the top right corner of the screen, and confirm the reboot

action.

Run the following scripts on the servers running Tripo:

stopsh

cleansh

start.sh

Start the SDC site by running the following command on the relevant servers:

crm node online

5.2 Rolling Back an EMS Site Upgrade to Release 4.4 before Activating the Backup Array

This procedure is used to roll back an EMS site upgrade, before the backup array was

activated. This procedure uses the backup array as the master copy.



Performing an Upgrade Rollback to 4.4 Rolling Back an EMS Site Upgrade to Release 4.4 before Activating the Backup Array



Stop the applications running on the EMS site by running the following

command on the relevant servers:

crm node standby

Re-mirror the two arrays using the backup array:


following steps:




iv. Press F5 (for Gen8) or F10 (for Gen9) to start the “HP Storage Controller

Administrator”.














Performing an Upgrade Rollback to 4.4 Rolling Back an EMS Site Upgrade to Release 4.4 after Activating the Backup Array or a Standalone SDC Site Upgrade





action.

Start the EMS site by running the following command on the relevant servers:

crm node online

5.3 Rolling Back an EMS Site Upgrade to Release 4.4 after Activating the Backup Array or a Standalone SDC Site Upgrade

This procedure is used to roll back two types of upgrades:

▪ An EMS site upgrade, after the backup array was activated.

▪ A standalone SDC site upgrade, after the backup array was activated.

This procedure uses the backup array as the master copy.



Enter “HP SSA” in the remote console.

Reboot the server.

At the prompt, enter “HP Storage Controller Administrator”.

Press F5 (for Gen8) or F10 (for Gen9) to start the “HP Storage Controller

Administrator”.


steps:

Under "Array Controller(s)" select the active array.



Performing an Upgrade Rollback to 4.4 Rolling Back an EMS Site Upgrade to Release 4.4 after Activating the Backup Array or a Standalone SDC Site Upgrade


From Actions list - Click "Configure"

Under "Controller Devices", select "Arrays"

Under "Arrays", select the "Active Array".

Under "Actions" select "Re-Mirror Array" and select Backup array as a source.

Note: Use the following command to monitor the status of the re-mirroring:

hpssacli ctrl all show config

Restore the original VG name, that was modified during the upgrade, by performing the

following steps:

Reboot the server and direct it to a RH recovery media.

i. In the first Rescue dialogue, select Skip, so that filesystems will not be

mounted.

ii. In the next dialog window, select Start Shell.

iii. At the shell prompt, running the following command to rename the VG:

vgrename vg1_clone vg1

The volume group “vg1_clone” is successfully renamed to “vg1”.

Unmount the ISO and reboot the server.



Performing an Upgrade Rollback to a Previous CF Rolling Back an SDC Site Upgrade to a Previous CF


6. Performing an Upgrade Rollback to a Previous CF

This upgrade process includes the option to roll back an upgrade that has encountered

errors.

▪ To roll back an upgrade of an SDC site managed by an EMS site, follow the Rolling

Back an SDC Site Upgrade to a Previous CF procedure.

▪ To roll back an upgrade of an EMS site before activating the backup array, follow the

Rolling Back an EMS Site Upgrade to a Previous CF before Activating the Backup

Array procedure.

▪ To roll back an upgrade of an EMS site after activating the backup array, follow the

Rolling Back an EMS Site Upgrade to a Previous CF after Activating the Backup

Array procedure.

6.1 Rolling Back an SDC Site Upgrade to a Previous CF This procedure is used to roll back an upgrade of an SDC site that is managed by an EMS

site. This procedure uses the backup array as the master copy.


Stop all traffic on the SDC site.

Stop the applications running on the SDC site and the EMS site by running the

following command on the relevant servers:

monit stop all

Run the following commands on each EMS site server to delete the SDC site

data:

cd /data/backup

mv <SDC rolled back site name> /var/tmp/





Start the EMS site.

On the SDC site, re-mirror the two arrays using the backup array:


following steps:




iv. Select F5 (for Gen8) or F10 (for Gen9) to start the “HP Storage Controller

Administrator”.



















action.

Run the following scripts on the servers running Tripo:

Note: The Tripo environment must be active before applying these scripts. You can do

so as follows:

su - traffix


. DefEnv Tripo

stopsh

cleansh

start.sh

Start the SDC site by running the following command on the relevant servers:

monit start all

Note: After rolling back an SDC site that is managed by an EMS site, the EMS global

configuration parameters will be distributed to the local SDC site. Any local configuration

changes made on the local SDC site after the upgrade and before the rollback will be

deleted from both the local SDC site and from the EMS site that manages it.

When a local SDC site is managed by an EMS, you need to revert the SDC

version saved in the EMS Cassandra to the original SDC version.

After the rollback, log in to Cassandra and run the following commands on one of

the Casandra hosted servers:

SELECT * FROM statusflow.appflow ;





SELECT * FROM statusflow.flow ;

Run the following commands with the following relevant parameters on each

database row in the app flow table and in the flow table, respectively:

i. Update command for each database row (meaning for each SDC

component/application) in the app flow table with the following

parameters:

Note: You need to run this command for each SDC component/application per

VM.

UPDATE statusflow.appflow SET "version" = '<previous SDC version>'

WHERE "siteId"= '<SDC site name>' AND "vmName"= '<VM name>'

AND "appType"= '<SDC component/application>' AND "appName"=

'<SDC component/application name>' ;

▪ version = previous SDC version

▪ siteID = SDC site name

▪ vmName = VM that is associated with the appType

▪ appType = the SDC component/application type (such as Web UI, CPF)

▪ appName = name of a SDC component/application type

ii. Update command for the flow table:

UPDATE statusflow.flow SET "current version" = '<current SDC

version>' WHERE "siteId" = '<SDC site name>' AND "flowType" =

'statusApi';

▪ currentVersion = current SDC version

▪ siteID = SDC site name



Performing an Upgrade Rollback to a Previous CF Rolling Back an EMS Site Upgrade to a Previous CF before Activating the Backup Array


▪ flowType = statusApi

Upon completing this step, the SDC version saved in the EMS Cassandra is

reverted back to the original SDC version.

6.2 Rolling Back an EMS Site Upgrade to a Previous CF before Activating the Backup Array

This procedure is used to roll back an EMS site upgrade, before the backup array was



Stop the applications running on the EMS site by running the following

command on the relevant servers:

monit stop all

Re-mirror the two arrays using the backup array:


following steps:




iv. Select F5 (for Gen8) or F10 (for Gen9) to start the “HP Storage Controller

Administrator”.







Performing an Upgrade Rollback to a Previous CF Rolling Back an EMS Site Upgrade to a Previous CF after Activating the Backup Array












action.

Start the EMS site by running the following command on the relevant servers:

crm start all

6.3 Rolling Back an EMS Site Upgrade to a Previous CF after Activating the Backup Array

This procedure is used to roll back an EMS site upgrade, after the backup array was





Enter “HP SSA” in the remote console.

Reboot the server.

At the prompt, enter “HP Storage Controller Administrator”.



Performing an Upgrade Rollback to a Previous CF Rolling Back an EMS Site Upgrade to a Previous CF after Activating the Backup Array


Select F5 (for Gen8) or F10 (for Gen9) to start the “HP Storage Controller

Administrator”.

Define and activate the active array as the master array by performing the

following steps:

Under "Array Controller(s)" select the active array.

From Actions list - Click "Configure"

Under "Controller Devices", select "Arrays"

Under "Arrays", select the "Active Array".

Under "Actions" select "Re-Mirror Array" and select Backup array as a source.

Note: Use the following command to monitor the status of the re-mirroring:

hpssacli ctrl all show config

Restore the original VG name, that was modified during the upgrade, by

performing the following steps:

Reboot the server and direct it to a RH recovery media.

i. In the first Rescue dialogue, select Skip, so that filesystems will not be

mounted.

ii. In the next dialog window, select Start Shell.

iii. At the shell prompt, running the following command to rename the VG:

vgrename vg1_clone vg1

The volume group “vg1_clone” is successfully renamed to “vg1”.

Unmount the ISO and reboot the server.



Mapping the Interfaces between Releases


Appendix A: Mapping the Interfaces between Releases The networking definitions previously defined in the SDC 4.4 release may be modified for



This section describes how to create a table of the networking changes. s


Run the following script on each server in the existing SDC 4.4 site:

If the upgraded server will use a management network, run the following script:

/opt/traffix/scripts/showNetMap.sh

If the upgraded server will not use a management network, run the following script:

/opt/traffix/scripts/showNetMap.sh 1



server:

CURRENT NEW HWADDR









Mapping the Interfaces between Releases


eth2 eth6 ac:16:2d:7d:11:48


eth10 eth8 ac:16:2d:7d:11:49

eth11 eth9 ac:16:2d:7d:11:4d

eth12 eth10 ac:16:2d:7d:11:4a

eth13 eth11 ac:16:2d:7d:11:4e





Appendix B: Port Settings Used by the SDC During an upgrade, a set of ports was enabled to ensure communication both between the

different SDC components within the deployment, and between the SDC components and

the necessary network elements.

This section describes the ports that have been validated for use by the SDC.

B.1 EMS Site Internal Ports

Table 22: EMS Internal Ports

Transport Protocol Port Network Description

TCP 2812 IC Monit

TCP 8089 MGMT Splunk Indexer

TCP 8189 MGMT Splunk Master license server

TCP 8289 MGMT Splunk Search Head

TCP 9888 MGMT Splunk Indexer

TCP 10030 MGMT Communication from the NMS

Agent to the Splunk Forwarder

TCP 13868 IC Traffic load balancing between

the FEP and CPF instances

TCP 61616 IC Communication between the

configuration manager and the

SDC components

TCP 61657 IC Web UI Communication on

cluster






UDP 161 IC SNMP GET functions provided

by OS snmpd service

UDP 162 IC SNMP traps listener from the

UDP 1162 IC OS trap daemon listener

TCP 7000 MGMT Cassandra Database inter-site

communication


communication

TCP 7199 MGMT Cassandra JMX monitoring

inter-site communication

TCP 9042 MGMT Cassandra client

B.2 EMS Site External Ports

Table 23: EMS External Ports

Transport

Protocol

Port In/Out Network Description

TCP 22/443 In MGMT SSH remote

consoles

TCP 80 In MGMT HP Blade System

web consoles

UDP 123 Out OAM NTP Process

UDP 514 Out OAM Syslog Process

UDP 1161 Out MGMT For External

EMS Statistics

Analysis





Transport

Protocol


UDP User-defined

Ports (and IPs)

Out MGMT Trap Forwarding:

For External

EMS Trap

listeners

TCP 3868 In/Out H-TCP Inter-site

communication

link for geo-

redundancy

SCTP 3868 In/Out H-SCTP-A Primary SCTP

path for domestic

traffic

SCTP 3868 In/Out H-SCTP-B Secondary SCTP

path for domestic

traffic

TCP 4505/6 In/Out MGMT Salt Master

TCP 8000 In/Out MGMT Salt API

TCP 8080/ 8443 In MGMT SDC web console

(Web UI)

TCP 8300 In OAM Splunk Search

reports loaded in

the Web UI

TCP 8200 In OAM Splunk Indexer

reports loaded in

the Web UI

TCP 8100 In OAM Splunk Master

TCP 8089 Out OAM Splunk Search

for the data in the

Indexer





Transport

Protocol


TCP 10045 Out OAM Communication

between the site's

Splunk

Forwarder and

the EMS's Splunk

Indexer. Also for

synchronizing

between EMS

Splunk Indexers.

UDP 10031 In OAM Communication

of Tracing from

the site Splunk

Forwarder to the

EMS Indexer

UDP 10032 In OAM Communication

(TDRs) from the

site Splunk

Forwarder and to

the EMS Indexer

TCP 10040 Out MGMT NMS Agent to

NMS Manager

for system status

synchronization

TCP 61617 In MGMT Communication

between the EMS

and the SDC

servers for new

configuration

propagation





Transport

Protocol


TCP 7000 In/Out MGMT Cassandra

Database inter-

site

communication

TCP 7001 In/Out MGMT Cassandra

Database inter-

site

communication

TCP 7199 In/Out MGMT Cassandra JMX

monitoring inter-

site

communication

TCP 9042 In/Out MGMT Cassandra client

B.3 SDC Site Internal Ports

Table 24: SDC Internal Ports


TCP 2812 IC Monit

TCP 61616 IC Communication between the

configuration manager and the

SDC components

TCP 13868 IC Traffic load balancing between

the FEP and the CPF instances

TCP 11812 IC RADIUS listening port

between the FEP and the CPF






TCP 18080 IC HTTP listening port between

the FEP and the CPF

TCP 13386 IC GTP listening port between the

FEP and the CPF

TCP 1389 IC LDAP listening port between

the FEP and the CPF

TCP 4444 IC NMS to CPF communication

port

TCP 23210 IC Tripo - CPF connection to

Tripo

TCP 43211 IC Tripo – inter-site connection

TCP 23212 IC Tripo - connection between

Tripo mates within the same

site

TCP 10030 MGMT Communication from the NMS

Agent to the Splunk Forwarder

TCP 61627 IC Default configuration REST

communication


communication


communication NMS Agent

TCP 61657 IC Default configuration rest

communication - UI

UDP 4545 IC Port prefix is 4545 and the

postfix is the UID of the CPF

or FEP (4545 + UID)






TCP 5555 MGMT Tripo Web statistics


communication


communication

TCP 7199 MGMT Cassandra JMX monitoring

inter-site communication

TCP 9042 MGMT Cassandra client

B.4 SDC Site External Ports

Table 25: SDC External Ports

Transport Protocol Port In/Out Network Description

TCP 4505/6 In/Out MGMT Salt Master

TCP 8000 In/Out MGMT Salt API

TCP 8080/8443 In MGMT SDC web console (Web

UI)

TCP 80 In MGMT HP Blade System web

consoles

UDP 162 Out MGMT SNMP traps toward the

EMS or third party NMS

servers

TCP 3868 In/Out H-TCP Inter-site

communication link for

geo-redundancy

SCTP 3868 In/Out H-SCTP-A Primary SCTP path for

domestic traffic






SCTP 3868 In/Out H-SCTP-B Secondary SCTP path

for domestic traffic

TCP 61617 In MGMT Communication between

the EMS and the SDC

servers for new

configuration

propagation (internal

and external data)

TCP 10045 Out MGMT Communication between

the SDC site Splunk

Forwarder and the

EMS's Splunk Indexer

UDP 10031 Out OAM Communication of

Tracing from the site

(CPF) to the site Splunk

Forwarder and to the

EMS Indexer

UDP 10032 Out OAM Communication (TDRs)

from the site (CPF) to

the site Splunk

Forwarder and to the

EMS Indexer

TCP 22/80/443/623/17990/

17988

In MGMT HP iLO4 management

consoles and virtual

media

TCP 10030 Out OAM NMS Agent

UDP 123 Out OAM NTP Process

UDP 514 Out OAM Syslog Process

TCP 8089/8090 In/Out OAM Splunk management






TCP 7000 In/Out MGMT Cassandra Database

inter-site

communication

TCP 7001 In/Out MGMT Cassandra Database

inter-site

communication

TCP 7199 In/Out MGMT Cassandra JMX

monitoring inter-site

communication

TCP 9042 In/Out MGMT Cassandra client

B.5 HP Integrated Lights-Out (iLO) Port Settings

The following information is not specific to SDC, but relates to relevant ports configured

on different servers.

Table 26: HP iLO Ports

Transport Protocol Port iLO Function

CP 22 Secure Shell (SSH)

TCP 80 Web Server Non-SSL

TCP 443 Web Server SSL

TCP 3389 Terminal Services

TCP 17988 Virtual Media

TCP 9300 Shared Remote Console

TCP 17990 Console Replay



Glossary


Glossary The following tables list the common terms and abbreviations used in this document.

Table 27: Common Terms

Term Definition

Answer A message sent from one Client/Server Peer to the other

following a request message

Client Peer A physical or virtual addressable entity which consumes AAA

services

Data Dictionary Defines the format of a protocol’s message and its validation

parameters: structure, number of fields, data format, etc.

Destination Peer The Client/Server peer to which the message is sent

Geo Redundancy A mode of operation in which more than one geographical

location is used in case one site fails

Master Session The session for which the routing selection is performed based

on the routing rules (Slave Sessions are applied with routing

rules inherited from the Master Session)

Orchestrator A workflow management solution to automate the creation,

monitoring, and deployment of resources in your environment

Origin Peer The peer from which the message is received

Pool A group of Server Peers

QCOW2 A file format for disk image files

RADIUS Remote Authentication Dial In User Service

REST Representation of a resource between a client and server

(Representational State Transfer)

Request A message sent from one Client/Server peer to the other,

followed by an answer message

RPM RPM Package Manager



Glossary


Term Definition

Salt-API Manages and communicates between an Orchestrator and

network master and minion servers

SDC Site The entire list of entities working in a single site

Server Peer A physical or virtual addressable entity which provides AAA

services

Session An interactive information interchange between entities

Slave (Bound)

Session

A session which inherits properties from a master session

SLRF Site An SLRF site including the Provisioning Manager

Transaction A request message followed by an answer message

Tripo Session data repository

vCenter Vmware Virtual Infrastructure tool for centralized management

of multiple hypervisors and enabling functionalities

Virtual Server A binding point used by SDC to communicate with the Remote

Peers (Clients and Servers)

Table 28: Abbreviations

Term Definition

AAA Authentication, Authorization and Accounting

ACL Access Control List

AF Application Function

API Application Programming Interface

AVP Attribute Value Pair

CLI Command Line Interface

CPF Control Plane Function

DEA Diameter Edge Agent



Glossary


Term Definition

DRA Diameter Routing Agent

EMS Site Element Management System Site

FEP-In In-Front End Proxy

FEP-Out Out-Front End Proxy

HA High Availability

HSS Home Subscriber Server

HTTP Hypertext Transfer Protocol

IaaS Infrastructure as a Service

IMS IP Multimedia Subsystem

JMS Java Message Service

KPI Key Performance Indicator

LDAP Lightweight Directory Access Protocol

LTE Long Term Evolution

MME Mobility Management Entity

NGN Next Generation Networking

Node Physical or virtual addressable entity

OAM Operation, Administration and Maintenance

OCS Online Charging System

OVF Open Virtualization Format

PCEF Policy and Charging Enforcement Function

PCRF Policy and Charging Rules Function

PLMN Public Land Mobile Network

SCCP Signaling Connection Control Part



Glossary


Term Definition

SCTP Stream Control Transmission Protocol

SDC Signaling Delivery Controller

SLRF Subscriber Location and Routing Function

SNMP Simple Network Management Protocol

SS7 Signaling System No. 7

TCP Transmission Control Protocol

TLS Transport Layer Security

UDP User Datagram Protocol

UE User Equipment

URI Universal Resource Identification.

VIP Virtual IP

VM Virtual Machine

VNFC Virtualized Network Function Component

VPLMN Visited Public Land Mobile Network

Web UI Web User Interface

WS Web Service

Signaling Delivery Controller Bare Metal System … · Signaling Delivery Controller Bare Metal System Upgrade ... F5 Signaling Delivery Controller Bare Metal System Upgrade Guide

Documents

Signaling Delivery Controller Bare Metal System … · Signaling Delivery Controller Bare Metal System Upgrade ... F5 Signaling Delivery Controller Bare Metal System Upgrade Guide