LDom

Brian Leonard, OpenSolaris Technical Marketing Engineer

Version 2.0 | Last updated: 06/18/09

HOW to CONFIGUREa LOGICAL DOMAIN

> OpenSolaris™ How To Guides

ContentsOverview Page 1

Configuring a Solaris 10 Guest Domain on OpenSolaris Page 1 Step 1: Check the Firmware Page 1

Step 2: Install the Logical Domains Manager Page 1

Step 3: Free Up Resources for the Guest Domain Page 2

Step 4: Create Services Needed to Support the Guest Domains Page 7

Step 5: Configure the Guest Domain Page 8

Step 6: Create a Virtual Boot Disk for the Guest Domain Page 8

Step 7: Create a Virtual Disk for the Guest Operating System Installation DVD Page 9

Step 8: Define a Virtual Network Interface Page 9

Step 9: Allocate Resources to the Guest Domain Page 9

Step 10: Log Into the Guest Domain Page 11

Step 11: Install Solaris 10 in the Guest Domain Page 11

Step 12: Operate the Guest Domain Page 12

For More Information Page 13

About This OpenSolaris How To GuideThis Logical Domain How To guide is intended to help OpenSolaris system adminstrators configure their SPARC CMT

machines with logical domains (LDoms). Users are guided step-by-step through the process of installing the logical

domain manager and configuring a guest domain.

For more information about LDoms, see http://opensolaris.org/os/community/ldoms/

OverviewLogical Domains (LDoms) allow you to virtualize OpenSolaris by creating isolated environments in which you can run other

operating systems that are LDom aware, namely Solaris and OpenSolaris. The virtualization features provided by LDoms

are designed into the SPARC CMT microprocessors and are therefore only available on machines containing either the

UltraSPARC T1 (32 threads), UltraSPARC T2 (64 threads), and UltraSparc T2+ (64 threads) chips.

To begin taking advantage of LDoms, the Logical Domains Manager software needs to first be installed which then runs

OpenSolaris in a “paravirtualized” state, meaning OpenSolaris is now aware of the virtualization layer beneath it. At this

point, there's a single domain, the “primary” domain, which is ready to provide virtualization services, such as networking

and disk i/o, to guest domains.

Before guest domains can be configured, resources such as CPUs and memory need to be removed from the primary

domain so that they can later be assigned to the guest domain. As part of the guest domain's configuration, a virtual disk

(or disks) needs to be set up. These virtual disks can map either to an actual disk or to a single file, which is the approach

that is used in this How To guide.

Once created, the guest domain still needs an operating system. As stated above, this can be either Solaris or OpenSolaris.

Solaris 10 is used as the guest operating system for this guide.

Configuring a Solaris 10 Guest Domain on OpenSolarisStep 1: Check the FirmwareAs mentioned above, the SPARC CMT microprocessors have virtualization features built into their firmware. Before

installing the Logical Domains Manager, make sure you have the latest firmware installed. You can check the latest firware

revs at http://docs.sun.com/source/820-4914-10/chapter1.html#d0e419. See the "Required Software to Enable LDoms 1.1

Features" table and the "Required System Firmware Patches" table.

The exact steps for checking and updating your firmware vary depending on your actual hardware. Please reference

http://docs.sun.com/app/docs/prod/coolthreads.srvr#hic for the specific steps that match your hardware.

Step 2: Install the Logical Domains ManagerBefore beginning, run the uname -m to verify your processor supports LDoms. The proper result will be sun4v.

The Logical Domans Manger is available in the IPS repository:

Install it as follows:

Logical Domain How To Guide

1OpenSolaris Operating System opensolaris.com

jack@dcsw-t5240:~$ uname -m

sun4v

jack@dcsw-t5240:~$ pkg info -r ldomsmanager

Name: system/ldoms/ldomsmanager

Summary: Logical Domains Manager

Category: System/Virtualization

State: Not installed

Publisher: opensolaris.org

Version: 1.1

Build Release: 5.11

Branch: 0.111

Packaging Date: Sat Apr 18 20:05:08 2009

Size: 8.16 MB

FMRI: pkg:/system/ldoms/[email protected],5.11-0.111:20090418T200508Z

Install it as follows:

Step 3: Free Up Resources for the Guest DomainOnce the Domain Manager is installed, OpenSolaris is running in a domain - the control domain - which is named primary.

You can see it by running ldm list:

The control domain is responsible for managing all of the other domains on the system and initially has all of the system

resources allocated to it. Before we can create a guest domain, we need to steal resources from the primary domain. In

the output above you can see that we have 128 virtual CPUs and 130GB of memory. The T5240 has two T2+ processors,

each with 8 cores and each core having 8 threads which gives us the total of 128 virtual CPUs. Theorectically, we could

create 127 guest domains, however allocating an entire core to a single domain is generally considered best for

peformance. That's because the T2+ processor in the T5240 I'm using has 4 cores, each represented as a virtual CPU.

You can view all of the free devices on the system by running:

At this point, the command does not return a list because all the system resources are assigned to the control domain. Use

the -a option to see all of the devices including the domain to which they're assigned:

OpenSolaris Operating System opensolaris.com

2

jack@dcsw-t5240:~$ pfexec pkg install ldomsmanager

DOWNLOAD PKGS FILES XFER (MB)

Completed 1/1 40/40 3.78/3.78

PHASE ACTIONS

Install Phase 74/74

PHASE ITEMS

Indexing Packages 1/1

ldm list-devices

jack@dcsw-t5240:~$ ldm list-devices

VCPU

PID %FREE PM

0 0 no

1 0 no

2 0 no

3 0 no

4 0 no

5 0 no

6 0 no

7 0 no

8 0 no

9 0 no

10 0 no

11 0 no

jack@dcsw-t5240:/usr/bin$ ldm list

NAME STATE FLAGS CONS VCPU MEMORY UTIL UPTIME

primary active -n-c-- SP 128 130848M 0.2% 3h 41m

OpenSolaris Operating System opensolaris.com

3

12 0 no

13 0 no

14 0 no

15 0 no

16 0 no

17 0 no

18 0 no

19 0 no

20 0 no

21 0 no

22 0 no

23 0 no

24 0 no

25 0 no

26 0 no

27 0 no

28 0 no

29 0 no

30 0 no

31 0 no

32 0 no

33 0 no

34 0 no

35 0 no

36 0 no

37 0 no

38 0 no

39 0 no

40 0 no

41 0 no

42 0 no

43 0 no

44 0 no

45 0 no

46 0 no

47 0 no

48 0 no

49 0 no

50 0 no

51 0 no

52 0 no

53 0 no

54 0 no

55 0 no

56 0 no

57 0 no

58 0 no

59 0 no

OpenSolaris Operating System opensolaris.com

4

60 0 no

61 0 no

62 0 no

63 0 no

64 0 no

65 0 no

66 0 no

67 0 no

68 0 no

69 0 no

70 0 no

71 0 no

72 0 no

73 0 no

74 0 no

75 0 no

76 0 no

77 0 no

78 0 no

79 0 no

80 0 no

81 0 no

82 0 no

83 0 no

84 0 no

85 0 no

86 0 no

87 0 no

88 0 no

89 0 no

90 0 no

91 0 no

92 0 no

93 0 no

94 0 no

95 0 no

96 0 no

97 0 no

98 0 no

99 0 no

100 0 no

101 0 no

102 0 no

103 0 no

104 0 no

105 0 no

106 0 no

107 0 no

OpenSolaris Operating System opensolaris.com

5

108 0 no

109 0 no

110 0 no

111 0 no

112 0 no

113 0 no

114 0 no

115 0 no

116 0 no

117 0 no

118 0 no

119 0 no

120 0 no

121 0 no

122 0 no

123 0 no

124 0 no

125 0 no

126 0 no

127 0 no

MAU

ID CPUSET BOUND

0 (0, 1, 2, 3, 4, 5, 6, 7) primary

1 (8, 9, 10, 11, 12, 13, 14, 15) primary

2 (16, 17, 18, 19, 20, 21, 22, 23) primary

3 (24, 25, 26, 27, 28, 29, 30, 31) primary

4 (32, 33, 34, 35, 36, 37, 38, 39) primary

5 (40, 41, 42, 43, 44, 45, 46, 47) primary

6 (48, 49, 50, 51, 52, 53, 54, 55) primary

7 (56, 57, 58, 59, 60, 61, 62, 63) primary

8 (64, 65, 66, 67, 68, 69, 70, 71) primary

9 (72, 73, 74, 75, 76, 77, 78, 79) primary

10 (80, 81, 82, 83, 84, 85, 86, 87) primary

11 (88, 89, 90, 91, 92, 93, 94, 95) primary

12 (96, 97, 98, 99, 100, 101, 102, 103) primary

13 (104, 105, 106, 107, 108, 109, 110, 111) primary

14 (112, 113, 114, 115, 116, 117, 118, 119) primary

15 (120, 121, 122, 123, 124, 125, 126, 127) primary

MEMORY

PA SIZE BOUND

0x0 512K _sys_

0x80000 1536K _sys_

0x200000 94M _sys_

0x6000000 32M _sys_

0x8000000 96M _sys_

0xe000000 130848M primary

Notice in the output above that the system also has 16 MAUs, which are cryptographic Mathmatical Arithmetic Units that

are used to accelarate cryptographic processing. There's a MAU associated with each core and we'll need to deal with

those as well as we free up resources.

Free up 1 MAU for the guest domain by setting the control domain to 15 MAUs:

Free up 8 CPUs (1 core) for the guest domain by setting the control domain to 120 CPUs:

Free up 16,356 MBs of memory for the guest domain by setting the control domain to 114,492 MB:

Now list-devices shows the resources that have been made available for the guest domain:

OpenSolaris Operating System opensolaris.com

6

IODEVICE PSEUDONYM BOUND OPTIONSpci@400 pci_0 yes pci@500 pci_1 yes

ldm set-mau 15 primary

ldm set-vcpu 120 primary

jack@dcsw-t5240:~$ ldm set-memory 114492M primary

Initiating delayed reconfigure operation on LDom primary. All configuration

changes for other LDoms are disabled until the LDom reboots, at which time

the new configuration for LDom primary will also take effect.

jack@dcsw-t5240:~$ ldm list-devices

VCPU

PID %FREE

120 100

121 100

122 100

123 100

124 100

125 100

126 100

127 100

MAU

ID CPUSET

15 (120, 121, 122, 123, 124, 125, 126, 127)

MEMORY

PA SIZE

0x1c01c00000 16356M

Step 4 : Create Services Needed to Support the Guest DomainsVirtual disk and network services need to be established in the control domain to support the guest domains. Create the

virtual disk service named primary-vds:

Create the virtual network service. In this example it is named primary-vsw0 and maps to the physical NIC nxge0:

Create the Virtual Console Concentrator service. We will use this service to log into the domain. In this example the service

is named primary-vcc0:

View all the configured services using the list-services command:

Reboot the primary domain so the changes can take effect. Note, if you know you're going to be creating multiple guest

domains, allocate the minimum number of resources you need for the guest domain to minimize the need for future

reboots:

After reboot, you can view the new configuration for the primary domain, now using only 120 CPUs and 114492M of

memory:

Finally, enable the virtual network terminal server daemon, which provides console services to the guest domain:

OpenSolaris Operating System opensolaris.com

7

ldm add-vds primary-vds0 primary

ldm add-vsw net-dev=nxge0 primary-vsw0 primary

ldm add-vcc port-range=5000-5100 primary-vcc0 primary

pfexec reboot

...

svcadm enable vntsd

jack@dcsw-t5240:~$ ldm listNAME STATE FLAGS CONS VCPU MEMORY UTIL UPTIMEprimary active -n-c-- SP 120 114492M 0.1% 3m

jack@opensolaris:~$ ldm list-servicesVCC

NAME LDOM PORT-RANGEprimary-vcc0 primary 5000-5100

VSWNAME LDOM MAC NET-DEV DEVICE DEFAULT-VLAN-ID PVID VID MODE primary-vsw0 primary 00:14:4f:f9:21:7c nxge0 switch@0 1 1

VDSNAME LDOM VOLUME OPTIONS MPGROUP DEVICEprimary-vds0 primary

Step 5: Configure the Guest DomainAdd a new guest domain called myldom:

Allocate the MAU:

Allocate the 8 free CPUs to the guest domain:

Allocate the 16356 megabytes of free memory to the guest domain:

Step 6: Create a Virtual Boot Disk for the Guest DomainThe guest domain needs a virtual boot disk. The device can either be an actual disk or a file in a ZFS file system, which was

the option chosen for this How To guide:

Create a new file system:

Create a file to use as the boot disk for the domain:

Define a virtual disk service device named vol1 for the zdisk file and associate it with the primary domain:

Define a virtual disk for the guest domain named vdisk1 that uses the primary domain’s virtual disk service:

Set the auto-boot property to false so when we first start the domain it doesn't try to boot from this empty disk:

OpenSolaris Operating System oepnsolaris.com

8

ldm add-domain myldom

ldm set-mau 1 myldom

ldm set-vcpu 8 myldom

ldm set-memory 16356M myldom

pfexec zfs create rpool/myldom

pfexec mkfile 10G /rpool/myldom/zdisk

ldm add-vdsdev /rpool/myldom/zdisk vol1@primary-vds0

ldm add-vdisk vdisk1 vol1@primary-vds0 myldom

ldm set-var auto-boot\?=false myldom

Step 7: Create a Virtual Disk for the Guest Operating System Installation DVDEither Solaris or OpenSolaris can be installed in the guest domain. If you prefer to install OpenSolaris, you'll need to use

the Automated Installer (AI). For more information on the AI, see http://www.opensolaris.com/use/Auto_Installer.pdf.

For this How To guide Solaris 10 will be installed as the guest operating system.

Download the Solaris 10 on SPARC ISO from http://www.sun.com/software/solaris/get.jsp.

Then verify it's MD5 checksum against the one provided:

Define a virtual disk service device for the ISO file and associate it with the primary domain. The disk service device

in this example is named solarisdvd:

Define a virtual disk for the guest domain that uses the primary domain’s virtual disk service. The virtual disk in this

example is named dvd:

Step 8: Define a Virtual Network InterfaceDefine a virtual network interface for the guest domain that uses the primary domain's virtual network service. The virtual

network interface in this example is named vnet0:

Step 9: Allocate Resources to the Guest DomainThe guest domain is currently inactive:

Run the bind command to activate it:

Reduce the memory on the guest domain by 256M and try the bind again:

OpenSolaris Operating System opensolaris.com

9

jack@dcsw-t5240:~$ md5sum sol-10-u7-ga-sparc-dvd.iso

c72ad13efb20904251ed334f99f95bd6 sol-10-u7-ga-sparc-dvd.iso

ldm add-vdsdev ~/sol-10-u7-ga-sparc-dvd.iso solarisdvd@primary-vds0

ldm add-vdisk dvd solarisdvd@primary-vds0 myldom

ldm add-vnet vnet0 primary-vsw0 myldom

ldm bind myldom

Not enough free memory present to meet this request

Could not bind requested memory for LDom myldom

jack@dcsw-t5240:/rpool/myldom$ ldm list

NAME STATE FLAGS CONS VCPU MEMORY UTIL UPTIME

primary active -n-cv- SP 120 114492M 0.5% 41m

myldom inactive ------ 8 16356M

ldm set-memory 16100M myldom

ldm bind

Now its state shows as bound:

You can run the list command to see the full configuration of the new domain:

OpenSolaris Operating System opensolaris.com

10

jack@dcsw-t5240:~$ ldm list -l myldom

NAME STATE FLAGS CONS VCPU MEMORY UTIL UPTIME

myldom bound ----- 5000 8 16100M MAC 00:14:4f:fb:b7:bf

HOSTID

0x84fbb7bf

VCPU

VID PID UTIL STRAND

0 120 100%

1 121 100%

2 122 100%

3 123 100%

4 124 100%

5 125 100%

6 126 100%

7 127 100%

MAU

ID CPUSET

15 (120, 121, 122, 123, 124, 125, 126, 127)

MEMORY

RA PA SIZE

0x2000000 0x1c02000000 16100M

NETWORK

NAME SERVICE DEVICE MAC MODE PVID VID

vnet primary-vsw0@primary network@0 00:14:4f:fb:2e:06 1

DISK

NAME VOLUME TOUT DEVICE SERVER MPGROUP

vdisk1 vol1@primary-vds0 disk@0 primary

dvd solarisdvd@primary-vds0 disk@1 primary

VCONS

NAME SERVICE PORT

myldom primary-vcc0@primary 5000

jack@dcsw-t5240:/rpool/myldom$ ldm list

NAME STATE FLAGS CONS VCPU MEMORY UTIL UPTIME

primary active -n-cv- SP 120 114492M 0.2% 1h 8m

myldom bound ------ 5000 8 16100M

Step 10: Log Into the Guest DomainNotice from the output above that the virtual console is running at port 5000. From the control domain:

In another terminal window, start the guest domain:

Back in the guest domains console, you should find yourself at the OK prompt:

Step 11: Install Solaris 10 in the Guest DomainOur next step is to install Solaris. First we need to find the device alias for the dvd virtual disk we configured earlier:

Use the dvd device alias to boot and install the domain:

OpenSolars Operating System opensolaris.com

11

jack@dcsw-t5240:~$ ldm start myldom

LDom myldom started

T5240, No Keyboard

Copyright 2008 Sun Microsystems, Inc. All rights reserved.

OpenBoot 4.29.0.a, 512 MB memory available, Serial #83528876.

Ethernet address 0:14:4f:fa:8c:ac, Host ID: 84fa8cac.

{0} ok

{0} ok devalias

dvd /virtual-devices@100/channel-devices@200/disk@1

vidsk1 /virtual-devices@100/channel-devices@200/disk@0

vnet0 /virtual-devices@100/channel-devices@200/network@0

net /virtual-devices@100/channel-devices@200/network@0

disk /virtual-devices@100/channel-devices@200/disk@0

virtual-console /virtual-devices/console@1

name aliases

{0} ok boot /virtual-devices@100/channel-devices@200/disk@1

Boot device: /virtual-devices@100/channel-devices@200/disk@1 File and args:

SunOS Release 5.10 Version Generic_139555-08 64-bit

Copyright 1983-2009 Sun Microsystems, Inc. All rights reserved.

Use is subject to license terms.

Configuring devices.

Using RPC Bootparams for network configuration information.

Attempting to configure interface vnet0...

Skipped interface vnet0

Reading ZFS config: done.

Setting up Java. Please wait...

...

jack@dcsw-t5240:$ telnet localhost 5000

Trying 127.0.0.1...

Connected to localhost.

Escape character is '^]'.

Connecting to console "myldom" in group "myldom" ....

Press ~? for control options ..

At this point you run through a standard installation of Solaris 10. Some items to note, the installer will recognize the

virtual network interface, vnet0:

As well as the 10GB virtual disk that was created:

Step 12: Operate the Guest DomainWhen installation completes, the system will boot and can be operated as any other standalone Solaris 10 system.

Note, the system can also be halted from the primary domain. First, verify that the guest domain is active:

Then stop the domain:

Stopping the domain using ldm stop causes an orderly shutdown. To immediately “power off” the machine, use ldm stop -f.

OpenSolaris Operating System opensolaris.com

12

─ Host Name for vnet0 ─────────────────────────────────Enter the host name which identifies this system on the network. The name

must be unique within your domain; creating a duplicate host name will cause

problems on the network after you install Solaris.

A host name must have at least one character; it can contain letters,

digits, and minus signs (-).

Host name for vnet0 myldom

───────────────────────────────────────────────Esc-2_Continue Esc-6_Help

─ Select Disks ─────────────────────────────────────On this screen you must select the disks for installing Solaris software.

Start by looking at the Suggested Minimum field; this value is the

approximate space needed to install the software you've selected. For ZFS,

multiple disks will be configured as mirrors, so the disk you choose, or the

slice within the disk must exceed the Suggested Minimum value.

NOTE: ** denotes current boot disk

Disk Device Available Space

=============================================================================

[X] c0d0 10152 MB (F4 to edit)

Maximum Root Size: 10152 MB

Suggested Minimum: 6424 MB

────────────────────────────────────────────────Esc-2_Continue F3_Go Back F4_Edit F5_Exit F6_Help

jack@dcsw-t5240:~$ ldm list

NAME STATE FLAGS CONS VCPU MEMORY UTIL UPTIME

primary active -n-cv- SP 120 114492M 0.9% 20m

myldom active -n---- 5000 8 16100M 2.1% 9m

jack@dcsw-t5240:~$ ldm stop myldom

LDom myldom stopped

13OpenSolaris Operating System opensolaris.com

For More InformationFor more information about Logical Domains and OpenSolaris, check out the following URLs:

Description

Logical Domains OpenSolaris Community

OpenSolaris LDoms Discussion Forum

OpenSolaris Automated Installion Project

OpenSolaris Home Page

http://opensolaris.org/os/community/ldoms/

http://opensolaris.org/os/community/ldoms/discussions/

http://opensolaris.org/os/project/caiman/auto_install/

http://www.opensolaris.com/

opensolaris.com

©2009 Sun Microsystems, Inc. All rights reserved. Sun, Sun Microsystems, the Sun logo, Solaris and OpenSolaris are trademarks or registered trademarks of Sun Microsystems, Inc. in the United States and other countries.SunWIN # 487538 Lit # SWWP11968-0 06/09

Sun Microsystems, Inc. 4150 Network Circle, Santa Clara, CA 95054 USA Phone 1-650-960-1300 or 1-800-555-9SUN Web sun.com