VMM for System Center 2012 General Troubleshooting Guide

Virtual Machine Manager (VMM) for System Center 2012 General Troubleshooting Guide

1

Released: Updated: September 8, 2011

This document is provided "as-is". Information and views expressed in this document, including URL and other Internet Web site references, may change without notice.

Some examples depicted herein are provided for illustration only and are fictitious.  No real association or connection is intended or should be inferred.

This document does not provide you with any legal rights to any intellectual property in any Microsoft product. You may copy and use this document for your internal, reference purposes. You may modify this document for your internal, reference purposes.

© 2011 Microsoft Corporation. All rights reserved.

Microsoft, Active Directory, Bing, Excel, Hyper-V, Internet Explorer, Silverlight, SQL Server, Windows, Windows Intune, Windows PowerShell, Windows Server, and Windows Vista are trademarks of the Microsoft group of companies. All other trademarks are property of their respective owners.

2

ContentsGeneral VMM Troubleshooting.........................................................................................................6

Description.......................................................................................................................................................................... 6

How to properly scope the issue................................................................................................................................6

Scoping Questions...................................................................................................................................6

Features Involved............................................................................................................................................................. 8

Underlying Technologies...............................................................................................................................................8

VMM Architecture Overview.................................................................................................................................................9

Introduction........................................................................................................................................................................ 9

General Troubleshooting Strategy.........................................................................................................10

VM Manager Log...................................................................................................................................10

Windows Event Logs..............................................................................................................................10

VMM Installation Troubleshooting..................................................................................................................................10

Installation Logs.....................................................................................................................................11

VMM Server Setup Logging...................................................................................................................11

VMM Agent Installation Logging.............................................................................................................................11

Windows Installer (MSI) Logging...........................................................................................................................11

Introduction......................................................................................................................................................................11

Viewing VMM Installation Logs...............................................................................................................................12

Troubleshooting Tools...........................................................................................................................14

Introduction......................................................................................................................................................................14

Data Collection Tools..............................................................................................................................................................14

Obtaining the legacy MPS Reporting utilities....................................................................................................14

Trace Collection........................................................................................................................................................................ 14

DebugView for Windows......................................................................................................................................................14

Introduction......................................................................................................................................................................14

Obtaining DebugView for Windows......................................................................................................15

Before Collecting a DebugView Trace....................................................................................................15

Collecting a DebugView Trace...............................................................................................................16

Trace32.exe................................................................................................................................................................................ 17

Introduction......................................................................................................................................................................17

3

Obtaining the Trace32.exe.........................................................................................................................................17

TextAnalysisTool.Net............................................................................................................................................................. 19

Introduction......................................................................................................................................................................19

SCVMM2012_MPSReports................................................................................................................................................... 20

Introduction......................................................................................................................................................................20

Installation........................................................................................................................................................................ 20

VMM Trace Collector – Managing and configuring tracing..........................................................................21

Add your machines to the Trace collector tool.................................................................................................21

Start and stop traces.....................................................................................................................................................21

Save your configuration for later............................................................................................................................22

VMM Trace Collector – Collecting traces.............................................................................................................22

VMM Trace Collector – Collecting MPS reports data......................................................................................23

Diagnostic Data location.............................................................................................................................................24

Filtering Trace Files...................................................................................................................................................... 25

Viewing Trace Files...................................................................................................................................................... 26

SQL Server Management Studio Express.......................................................................................................................27

Introduction......................................................................................................................................................................27

Obtaining SQL Server Management Studio Express.......................................................................................27

Advanced Troubleshooting....................................................................................................................29

Interpreting Job Failures...................................................................................................................................................... 29

Introduction......................................................................................................................................................................29

Where to begin................................................................................................................................................................ 29

Tools to Use.......................................................................................................................................................................29

Interpreting Job Failures.............................................................................................................................................29

Analyzing the Trace................................................................................................................................................................ 32

Introduction......................................................................................................................................................................32

WinRM Troubleshooting...................................................................................................................................................... 35

Test local WinRM functionality................................................................................................................................36

Test remote WinRM functionality...........................................................................................................................36

WMI Troubleshooting............................................................................................................................................................37

WMI Troubleshooting Tools......................................................................................................................................37

Windows Management Instrumentation Tester (wbemtest.exe).............................................................38

WMI Service Control Utility (winmgmt.msc).....................................................................................................39

4

WMIC command line.....................................................................................................................................................40

WMI Error Constants....................................................................................................................................................41

BITS Troubleshooting............................................................................................................................................................ 42

Introduction......................................................................................................................................................................42

Verify BITS outside of VMM.......................................................................................................................................42

BITS Compact Server....................................................................................................................................................43

BITS Traces....................................................................................................................................................................... 44

Useful KB Articles and Blogs.....................................................................................................................................47

5

General VMM Troubleshooting

DescriptionIn a high-level overview, troubleshooting Virtual Machine Manager (VMM) for System Center 2012 can be divided into the following categories:

1. How to properly scope the issue.

2. What are the VMM features involved?

3. What are the underlying technologies that VMM relies upon?

How to properly scope the issueBefore work can begin on troubleshooting any VMM issue, you must properly identify or scope the issue. This involves asking the right questions, listening and sometimes interpreting the answer as well as collecting some basic information from the customer.

Scoping QuestionsScoping questions are questions designed to help determine the actual issue.

Note:This list is targeted at common questions that should be asked in all VMM cases. Additional scoping is required once a specific problem area has been identified.

What is the installed version of VMM including build?

Is this a clean install of VMM or an update from a previous version or beta?

What is the exact syntax of the error message?

When does the error occur? During what VMM operation?

What is the installed version of SQL?

Is SQL local or remote?

Scope the issue precisely and thoroughly

Before deciding on what will be addressed as the issue, make sure you have asked yourself questions such as these:

Is the issue reproducible?

When exactly did it begin, and what was changed prior to that time?

6

What exactly does not work? This means there is an expected outcome and something else is instead occurring (or not).

What constitutes a working scenario? Performing as it once did? Performing as expected for the first time?

Which features are affected? Which features are not affected?

Of the items that are not working as expected, what other technologies are dependent upon or related? This is your starting point.

Develop a plan

Test in various methods those items that have been identified as not working as expected. Until a conclusion or consensus is made regarding a feature, do not move on to other seemingly related issues. Stay on task.

Once a technology has been ruled out as working correctly, move forward, not back. If it is decided that a feature is working focus should be shifted to the next relevant technology.

Research relevant errors

Begin with general errors and review carefully all messages reported by the product. Error messages have improved significantly and often simply state the cause of the problem and how to fix it.

If there are very specific error codes (0x8XXXXXXX) search for these. General four digit errors are useful and can be referenced for general information in the error codes appendix, or online at TechNet, but eight digit errors are specific to the problem at hand. Use these at every opportunity.

Resource List

Knowledge Base (KB)

http://support.microsoft.com/kb/242450

System Center Content Search Gadget

http://gallery.live.com/LiveItemDetail.aspx?li=49e26ad0-113d-4f3d-a711-57f6530c75d9

TechNet - System Center Virtual Machine Manager 2008

http://technet.microsoft.com/en-us/library/cc917964.aspx (Direct)

http://technet.com/scvmm (Select ‘Library’ tab at top)

Official Microsoft Blogs

http://blogs.technet.com/<Name>

scvmm, m2, hectorl, virtualization, jonjor, mbriggs

7

Features InvolvedThe following features are supported in VMM for System Center 2012.

Underlying TechnologiesThe following list contains core underlying technologies that VMM depends upon in order to work correctly.

WS Management (WinRM)

WMI

BITS

DCOM

WCF

8

VMM Architecture Overview

IntroductionVMM is modular in design with the following three layers, joined by a common, well-documented interface. Before understanding where to begin when troubleshooting VMM, it is helpful to understand the underlying architecture to determine where to begin.

Each of the core operating system underlying technologies will be discussed in more detail later in this document.

9

General Troubleshooting StrategyAt this point, the issue has been properly scoped and a general understanding of the architecture has revealed underlying core technologies that might not be working correctly. The next steps are to use the available tools outlined below to analyze all involved computers, and to collect data for analysis.

VM Manager LogOne of the first places to look for errors or warnings is the VM Manager log. The VM Manager log can be found on the VMM Management Server in the Diagnostics node in Server Manager.

1. Open Server Manager and expand the Diagnostics node2. Select Event Viewer and then expand 3. Applications and Services and find the VM Manager log underneath

Scan this log for the presence of any Error messages. Correlate this to the date and time which the problem occurred.

Windows Event LogsIt is also a good idea to examine the Windows event logs for corresponding errors or failures. As often is the case, an underlying problem with the operating system or one of its core services will result or bubble up as an problem in VMM.

For example: The WMI service has failed to start properly or was set to disabled. WMI is one of the core services that VMM relies upon for communication with the vmmagent. If the WMI service is not working properly, many operations in VMM will fail as a direct result of this.

This could easily be determined by looking in the System or Application log for corresponding failures or errors that would indicate that the service is not running.

VMM Installation TroubleshootingThis section discusses VMM feature installation-related troubleshooting. The following topics are discussed in this section:

Installation Logs

o VMM Server Setup Logging

10

o VMM Agent Installation Logging

Windows Installer (MSI) Logging

Viewing VMM Installation Logs

Known Issues – Setup and Upgrade

Installation LogsVMM logs installation information for the main VMM Server as well as agent installation.

VMM Server Setup LoggingSeparate reports exist for the installation of each of the primary features of VMM. Installation logs are written, by default, to the following hidden folder on Windows Server 2008:

C:\ProgramData\VMMLogs

VMM Agent Installation LoggingLocal agent installation information is logged in the following hidden folders:

Windows Server 2003:

C:\Documents and Settings\All Users\Application Data\VMMLogs

AgentSetup.log

vmmAgent.msi_<m-d-yyy_hh-mm-dss>.log

vmmmsxml6setup_<m-d-yyy_hh-mm-dss>.log

vmmvssetup_<m-d-yyy_hh-mm-dss>.log

Windows Server 2008:

C:\ProgramData\VMMLogs

vmmAgent.msi_<m-d-yyy_hh-mm-dss>.log

Important:

If installation logging does not provide enough information to determine the cause of a failure, tracing may be enabled prior to starting the installation, using the VMM MPS Reports tool.

Windows Installer (MSI) Logging

IntroductionVMM features that are provided as Windows Installer packages (.msi files), including agent installation packages, are installed using the Windows Installer service (MSI). When installing an MSI package, such as installing the VMM agent manually, you can enable logging using the following command:

msiexec /I <MSIPackageName.msi> /L*V <path\logfilename>.log

11

Host Agent install example:

msiexec /I “C:\Program Files\Microsoft System Center 2012\Virtual Machine Manager\agents\amd64\2.0.5007.0\vmmAgent.msi” /L*V c:\temp\vmmagent.log

Viewing VMM Installation Logs

When viewing the installation logs for the main VMM features, begin by viewing the log from the bottom, which lists the most recent activity.

More Information:When viewing logs, it is common to see errors reported as Carmine error. “Carmine” was the code name of VMM during its development process.

In most cases, the primary error text will be at the bottom of the log and will be indented from the rest of the lines as shown in the following figure.

Figure 1: Sample vmmServer.log file

12

As with any log-based troubleshooting, having a good understanding of what a successful installation log contains is valuable when trying to identify an issue versus expected exceptions. Fully understanding the problem scenario, then successfully reproducing the same steps to obtain a known-good log is often needed so a valid working versus non-working log analysis may be performed.

13

Troubleshooting ToolsIntroductionMany tools can be used to troubleshoot problems that occur in VMM. Some can be classified as data collection tools, some help with capturing traces, some convert trace files from ETL format to LOG. In the sections that follow, you'll learn how to collect and analyze data surrounding a problem with VMM.

Data Collection ToolsThe following tools are used to collect data surrounding a VMM problem.

Obtaining the legacy MPS Reporting utilitiesThe legacy Microsoft Product Support's Reporting Tools are available at the following location:

http://www.microsoft.com/downloads/details.aspx?FamilyID=cebf3c7c-7ca5-408f-88b7-f9c79b7306c0&DisplayLang=en

Trace Collection

DebugView for Windows

IntroductionDebugView for Windows is an application that lets you monitor debug output on your local system, or any computer on the network that you can reach via TCP/IP. It is capable of displaying both kernel-mode and Win32 debug output, so you don't need a debugger to catch the debug output your applications or device drivers generate, nor do you need to modify your applications or drivers to use non-standard debug output APIs.

970066 How to collect traces in System Center Virtual Machine Managerhttp://support.microsoft.com/default.aspx?scid=kb;EN-US;970066

Important:DebugView is generally only required when it is not possible to obtain an ETL trace through the MPSReport tool. DebugView does, however, provide information about technologies that are not directly related to VMM but that may have an effect on VMM behavior.

14

Note:‘SCTrace.cmd’ is a script that automates the trace collection process by performing the steps below. This utility may save time and make the trace capturing process more user friendly for customers. The utility can be found at the following location:

http://blogs.technet.com/jonjor/archive/2009/04/07/scvmm-tracing-made-easy.aspx

Obtaining DebugView for WindowsDebugView for Windows can be downloaded from the Windows Sysinternals site at the following link: http://technet.microsoft.com/en-us/sysinternals/bb896647.aspx

Before Collecting a DebugView Trace1. Install DebugView on the VMM server, the host in question, and/or the Web server (for

troubleshooting self-service portal issues).

2. Save the following code into a text file and name it "odsflags.cmd":

@echo offecho ODS control flags - only trace with set flags will go to ODS

if (%1)==() goto :HELPif (%1)==(-?) goto :HELPif (%1)==(/?) goto :HELP

echo Setting flag to %1...reg ADD "HKLM\SOFTWARE\Microsoft\Windows NT\CurrentVersion\Tracing\Microsoft\Carmine" /v ODSFLAGS /t REG_DWORD /d %1 /fecho Done.goto :EXIT

:HELPecho Usage: odsflags [flag], where flag isecho TRACE_ERROR = 0x2,echo TRACE_DBG_NORMAL = 0x4,echo TRACE_DBG_VERBOSE = 0x8,echo TRACE_PERF = 0x10,echo TRACE_TEST_INFO = 0x20,echo TRACE_TEST_WARNING = 0x40,echo TRACE_TEST_ERROR = 0x80,

:EXIT

3. Save the following code into a text file and name it "odson.reg":

Windows Registry Editor Version 5.00

[HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\Windows NT\CurrentVersion\Tracing\Microsoft\Carmine]"ODS"=dword:00000001

4. Save the following code into a text file and name it "odsoff.reg":

15

Windows Registry Editor Version 5.00

[HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\Windows NT\CurrentVersion\Tracing\Microsoft\Carmine]"ODS"=dword:00000000

5. Copy the three files created above, odsflags.cmd, odson.reg, and odsoff.reg, to the machines that DebugView will be used on.

6. Run the following commands at an elevated Command Prompt:

odson.regodsflags.cmd 255

(If you need to collect traces for both VMM Server and the host or the Web server, make sure to run these commands on all of these computers).

Collecting a DebugView Trace1. Open DebugView as an administrator and ensure that both Capture Win32 and Capture

Global Win32 are selected on the Capture menu item. You should be able to see tracing from the VMM features showing up in DebugView. (If you need to collect traces for both VMM management server and the host, make sure to do these steps on all of these computers).

Figure 2: DebugView.exe Capture menu

2. Restart the Virtual Machine Manager Service on the VMM management server by running the following commands at an elevated command prompt:

net stop vmmservicenet start vmmservice

3. Restart the Virtual Disk Service by running the following commands at an elevated command prompt:

net stop vdsnet start vds

16

4. Restart the Virtual Machine Manager Agent service on the host by running the following commands at an elevated command prompt:

net stop vmmagentnet start vmmagent

5. Restart the IIS service on the Web server by running the following command at an elevated command prompt:

iisreset

6. Reproduce the issue.

7. After reproducing the issue, save the output from the DebugView to a text file for analysis.

Important: Turn off tracing after collecting the data by running the following command at a Command Prompt:odsoff.reg

The format of a DebugView trace differs from that of an ETL trace since DebugView captures all Win32 application activity and, if selected, Kernel mode activity. If saved as a text file, it may be opened in other tools such as TextAnalysisTool.net or Trace32.

Trace32.exe

IntroductionTrace32.exe, an executable found in System Center Configuration Manager 2007, can quickly open very large trace files and will automatically highlight lines with apparent errors. This tool will allow you to quickly open very large files and locate errors visually. Built-in are tools for filtering based on various parameters as indicated in the figure below.

Obtaining the Trace32.exeTrace32.exe can be found at the location below, or with Configuration Manager 2007 media:http://www.microsoft.com/downloads/en/details.aspx?displaylang=en&FamilyID=5a47b972-95d2-46b1-ab14-5d0cbce54eb8

17

Figure 3: Trace32 with error lines highlighted

Figure 4: Filter settings in Trace32

18

TextAnalysisTool.Net

Introduction TextAnalysisTool.NET was previously a Microsoft internal tool, but the author has made it available on his blog, http://blogs.msdn.com/delay/archive/2007/06/21/powerful-log-file-analysis-for-everyone-releasing-textanalysistool-net.aspx as well. One benefit of this utility is the speed in which it can open very large files, similar to Trace32.exe. What makes this tool of greater value are the filters, markers, and quick search it provides. A description and screenshot of the tool’s features from the author below.

Filters: Before displaying the lines of a file, TextAnalysisTool.NET passes the lines of that file through a set of user-defined filters, dimming or hiding all lines that do not satisfy any of the filters. Filters can select only the lines that contain a sub-string, those that have been marked with a particular marker type, or those that match a regular expression. A color can be associated with each filter so lines matching a particular filter stand out and so lines matching different filters can be easily distinguished. In addition to the normal "including" filters that isolate lines of text you DO want to see, there are also "excluding" filters that can be used to suppress lines you do NOT want to see. Excluding filters are configured just like including filters but are processed afterward and remove all matching lines from the set. Excluding filters allow you to easily refine your search even further.

Markers: Markers are another way that TextAnalysisTool.NET makes it easy to navigate a file; you can mark any line with one or more of eight different marker types. Once lines have been marked, you can quickly navigate between similarly marked lines - or add a "marked by" filter to view only those lines.

Find: TextAnalysisTool.NET also provides a flexible "find" function that allows you to search for text anywhere within a file. This text can be a literal string or a regular expression, so it's easy to find a specific line. If you decide to turn a find string into a filter, the history feature of both dialogs makes it easy.

19

Figure 5: TextAnalysisTool.NET

SCVMM2012_MPSReports

IntroductionThe VMM tracing tools were developed to provide a utility to manage, collect, and view various traces and diagnostic information in a VMM environment. It is made up of the VMM Trace Collector utility (mpsrpt_setup.exe) and the VMM Trace Viewer utility (TraceViewer.exe). The Trace Collector utility is used for managing and collecting traces, while the Trace Viewer utility is used for viewing VMM traces.

You can download the tool from here: https://connect.microsoft.com/site799/Downloads

You must register with the VMM for System Center 2012 Release Candidate program on Microsoft Connect to download the tool.

InstallationYou can install the VMM tracing tools by running the MPSRpt_Setup.exe file which is an Out of band (OOB) release. The installation will prompt for a destination path to save the files.

Tip:if you have a problem installing the application directly, try installing it from an elevated command prompt, using the command: “MPRRpt_Setup.EXE /Q /T:c:\mpsrpt /C” and check the log file for errors.

20

Note:In the above example, "c:\mpsrpt" is an example location. The location field is user defined and may be any location on the VMM Management server.

SCVMM2012_MPSrpt has a dependency on the Sysinternals tool PsExec which is a light-weight telnet replacement that lets you execute processes on other systems, complete with full interactivity for console applications, without having to manually install client software.

Important:PsExec must be located in the same folder as the SCVMM2012_MPSReports, or the c:\mpsrpt in the above example. PsExec, which is part of PStools may be downloaded at the following link: HYPERLINK "http://technet.microsoft.com/en-us/sysinternals/bb897553.aspx"http://technet.microsoft.com/en-us/sysinternals/bb897553.aspx

VMM Trace Collector – Managing and configuring tracingYou can manage your traces through the VMM Trace Collector tool, mpsrpt_setup exe. To manage your traces, perform the following steps.

Add your machines to the Trace collector toolAdd your VMM server, hosts, library servers, clients, and P2V machines to the tool using the “Add Machine” button and typing the machine name into the new node.

Manage and organize your machines into groups using the “Add Group” button, and typing the name of your new group into the new folder.

Add machines and sub groups to your groups by selecting the group in the tree, then clicking the “Add Machine” or “Add Group” button.

Start and stop tracesOnce you have your machines added and organized into groups, you can start and stop tracing on those machines by either right clicking on the groups or individual machines and selecting an option for which traces to enable.

21

You can disable traces in the same way.

Save your configuration for laterYou can save your configuration and load a saved configuration using the load and save buttons on the main toolbar.

When the tool loads, it loads the DefaultMachineConfig.mconfig configuration file. Replace this file if you would like a new default.

VMM Trace Collector – Collecting tracesTo collect traces on one or multiple machines, select the machine or group in the tree, right click and select an option under the Collect Trace context menu.

Select the “Advanced…” option to specify directory, session name, date ranges, and other options.

22

VMM Trace Collector – Collecting MPS reports dataTo collect additional diagnostic information that was originally in the MPSRPT tool, select the machine or group and click the “Generate Reports” button on the main toolbar.

This will bring up the Reports UI, where you can specify the diagnostic information you wish to collect. Select the diagnostic information you want to collect and click the generate button. The report collector uses the MPSRPT engine to collect these reports, so the same permissions must be applied to these machines for this functionality to work properly.

Important:

23

Diagnostic Data locationBy default, diagnostic data is collected to this location

C:\MpsRpt\CollectedTraces\<trace session name>, but this can be configured in the advanced options dialog.

24

Filtering Trace FilesThe trace has a new feature to summarize (filter) large log files based on messages, dates and feature.

25

Viewing Trace FilesThe trace (.ETL) logs created by the SCVMM2012_MPSReports tool are not human-readable and must be loaded into an application that is capable of interpreting them. The preferred tool for interpreting the trace logs is the TraceViewer.

The Trace Viewer (traceviewer.exe) has the following two primary functions:

Converts the default binary .ETL trace logs into .CAR files that can be viewed in both TraceViewer and other trace parsing tools such as the TextAnalysisTool.net.

Provides basic trace parsing.

To convert the ETL file into CAR format for analysis, launch TraceViewer and drag the trace file into the open pane. This will result in a prompt on where to save the CAR file.

Note:The default location is the same folder as the ETL or trace file.

Once the file has been saved, TraceViewer will open the converted ETL file for analysis.

26

SQL Server Management Studio Express

IntroductionMicrosoft SQL Server Management Studio Express is a freely-available, easy-to-use graphical management tool for managing instances of a SQL Server Database Engine created by any edition of SQL Server 2005. It may be used to view and edit the SCVMM-specific SQL database tables.

Obtaining SQL Server Management Studio ExpressSQL Server Management Studio Express is available externally at the following location:

http://www.microsoft.com/downloads/details.aspx?FamilyId=C243A5AE-4BD1-4E3D-94B8-5A0F62BF7796&displaylang=en

Installation instructions are provided on the download page.

27

Figure 6: Viewing TR_TaskTrail database

28

Advanced Troubleshooting

Interpreting Job Failures

IntroductionReviewing VMM traces has become so commonplace that we may tend to forget other information related to the failure. More important than the trace itself is the context of the failure. What was being performed at the time and what error message was recorded for the task? These other sources of information are your starting point. They are what make the trace valuable.

Reading a VMM trace is an art and a science. There are basic methods to follow and items to locate in the trace, but the only way to become proficient is practice, practice, practice. Perform actions on your own machines, gathering a trace at the same time, and then review the trace to learn how the actions you initiated are recorded. What follows are general best practices for resolving customer issues by using all data available.

Where to beginTo understand a failure you need information from various sources. You cannot begin blindly reading a trace, for example, and expect to get far. Prepare yourself with these items:

What job was being run at the time of the trace? A P2V, adding a host?

Which trace do you have? Depending on the job you will need traces from more than one system. A trace from the VMM management server will always be needed. If performing a P2V a trace will be required from the Source machine and Destination Host as well. You must also know which trace is from which machine.

Did the job fail or was it cancelled? The better answer is ‘failed’. A failed job will produce an error in the Admin Console along with a hex error code. This error code is the first item you will search for in the trace starting from the bottom, so it’s important to have.

Verify that the trace being reviewed was run during the time that the error occurred.

Tools to UseEssentially, Notepad.exe is all that is required to view a trace. Unfortunately, Notepad takes a long time to open the large traces created by VMM. In the examples that follow, TextAnalysisTool.Net will be used instead. A copy of all error codes returned by VMM should also be kept on hand for reference. A list of VMM 2008 R2 error codes is available here: http://social.technet.microsoft.com/wiki/contents/articles/virtual-machine-manager-vmm-2008-r2-error-codes.aspx

29

Interpreting Job FailuresWhen a job fails, select the ‘Jobs’ tab in the Admin Console and review the error recorded. There will be an error code, usually a 3 to 5 digit number. Below this will be more specific information that explains what went wrong in plain English (or localized language). There will also be a return code in hexadecimal format and a recommended action if available.

The error will begin with ‘0x’ followed by eight digits. Often this return code is related to WinRM and if the meaning of the return code is not already provided it can easily be determined. Take the example below:

Figure 7: Error from VMM Admininstrator Console

‘Error (2915),’ though specific, is really not specific enough. Perform a search of the VKB or the Internet and you will find numerous reasons for this error. To use this error effectively let’s dig deeper.

First, read the error as it is presented. ‘The WS-Management service cannot process the request. Object not found on the <servername> server.’ This error also provides a Recommended Action. All of this is useful information:

WS-Management is a reference to WinRM. So the issue is likely caused by a condition that led to a general communication failure, which bubbles up in VMM as a WinRM error. Checking the WinRM service on the indicated server would be a good starting place.

Object not found on <servername>. Whatever the name of the server provided, this is where you should focus your efforts.

The recommended action suggests that the agent be checked, and recommends rebooting <servername>. The only agent used in VMM is the VMM Agent or P2V Agent. A better action than rebooting the server indicated would be to check and restart the VMM Agent service.

From this simple error we have two action plans:

1. Verify the WinRM service on the remote server.

2. Verify the VMM Agent on the remote server.

30

Further, if all servers are reporting this error, it seems likely that the issue may in fact involve the VMM management server itself. Check the services mentioned on the VMM management server, and verify communication with simple WinRM tests explained later in this document under ‘WinRM Troubleshooting’.

Now let’s imagine two things being different regarding this error. First, let’s imagine that there was no explanation for the return code (not too difficult as it did report ‘Unknown error (0x80338000)), and that there was no recommended action. What now? First, look up the original 3 to 5 digit error code, in this case ‘2915.’ Searching ‘Error Codes_VMM R2’ we find the following:

2915 The WS-Management service cannot process the request. Object not found on the %ServerName; server.

Ensure that the agent is installed and running. If the error persists, reboot %ServerName; and then try the operation again.

This represents the code, the message, and the recommended action. So, in this case the recommended action was already provided in the Admin console message, but this is not always the case. Let’s move on.

Resolve the return code. As these are usually WinRM related, start there.

winrm helpmsg 0x80338000

This returns the following:

winrm helpmsg 0x80338000The WS-Management service cannot process the request. The service cannot find the resource identified by the resource URI and selectors.

Ok, maybe this is not the most useful error message, but it is a bit different than that provided in the Admin Console and may give you a few key terms that return a better result on the Internet. Also, notice that if you did not precede the eight digit code with a ‘0x’ nothing is returned. This is a ‘feature’ of winrm help.

Keep in mind that if winrm returns nothing for the error message, the error probably is not winrm related. There is one additional trick that can narrow down an error even more. Take the last four digits of an eight digit hex code and run this through ‘net helpmsg’. This is worth testing, but the results are not predictably useful.

A final comment on return codes; you may have noticed that most of the return codes you see begin with ‘0x8’. If the first four digits of the code begin with 8004, 8007, or 80005 (three zeros) you have a WMI related error. Don’t mistake this to mean the error is due to WMI, just that its origin can be determined. Using the table below we see that errors beginning with 80041xxx or 800440xx did in fact originate in WMI, and so WMI should be investigated. Errors beginning with 8007xxx, 80040xxx and 80005xxx originated elsewhere, although they were reported through WMI. Skip WMI and look further for the source of the error in most cases with these. This table is included in the WinRM and WMI appendix module of this training.

31

Tip:Occasionally there will be an error code beginning with ‘-2’ that is ten digits long. These can be converted into regular hexadecimal numbers by entering the number, minus sign included, into calc.exe while in decimal mode, then change the format to hex.-2147024809 for example becomes FFFFFFFF80070057. Just remove the first eight ‘F’s.

Table 1: Common ranges of WMI errors

Term Description

0x800410xx - 0x800440

Errors that originate in WMI itself.A specific WMI operation failed because of: An error in the request, for example, a WQL query fails or the

account does not have the correct permissions. A WMI infrastructure problem, such as incorrect CIM or DCOM

registration.

0x8007xxx Errors originating in the core operating system. WMI may return this type of error because of an external failure, for example, DCOM security failure.

0x80040xxx Errors originating in DCOM. For example, the DCOM configuration for operations to a remote computer may be incorrect.

0x80005xxx Error originating from ADSI (Active Directory Service Interfaces) or LDAP (Lightweight Directory Access Protocol), for example, an Active Directory access failure when using the WMI Active Directory providers.

Analyzing the Trace

IntroductionIf there is a trace available, the first thing to do is identify the task failure in the ‘Job’ view of the Admin Console. This represents the actual job failure and is your starting point. Begin at the bottom of the trace and search up for the hex error code (0x80041005 for example). It will be there if you are in fact looking at the right trace. Although both keywords below can be found in a trace, here’s a tip on determining what is being called:

If the keyword ‘ServerConnection’ is found, this is a Host making reference to an attempt to contact the VMM Server.

If the keyword ‘ClientConnection’ is found, this is a Server making reference to an attempt to contact a VMM Host.

Once you have found the hex return code in the trace you need to understand the structure of the trace. First, keep in mind that traces are asynchronous. Simply meaning that there are many jobs running at once, all being recording by the trace, and that the line right above the one you find of interest may be from an entirely unrelated task. Pay attention to the PID (Process ID) and TID (Thread ID) of the line you are on… and if there is a TaskID, write it down as this represents the job itself. As you move up through the trace beginning with the hex code you will likely run into an

32

exception. An exception represents a job failure and probably has the answer to the issue at hand. Exceptions are also easy to identify as they are indented and many lines begin with ‘at Microsoft.’. In the example below work your way up searching for ‘0x80338029’. The ‘TaskID’ is also visible. Finally, notice that the bottom three lines are not related to the exception as the PID and TID do not match that of the exception lines.

Figure 8: Trace example

Tip:Even though the word ‘exception’ will allow you to locate failures in a trace easily, not all exceptions are related to real issues. Exceptions involving ‘NPIV’ for example are numerous and can usually be ignored.

Walking up through the trace it is possible to isolate the exact function or operation being performed at the time the exception occurred. Often there is a corresponding remote WMI call being made that fails. These remote WMI calls are delivered as the payload of a WS-Man request, and then ran on the remote machine. Take note of these operations and attempt to reproduce outside of VMM. More information on this is below in the WinRM and WMI sections.

Isolating specific tasks in a traceEach VMM task and subtask is identified by a task ID. The task ID is a GUID assigned to a task when the task is built. If any subtasks are required to complete a primary task, then a separate subtask ID is assigned to the subtask. Background tasks, such as refresh operations, capacity

33

management also have specific task IDs assigned to them. When troubleshooting problems that are not related to a specific user-initiated task, it is important to determine the task ID for the background task.

Every task performed by VMM is tracked and stored via the following three key tables in the VMM database:

Audit Task Trail database

Task trail database

Subtask trail database

There are separate databases for storing individual task types, such as refresh operations, but these three databases can typically be used to identify all task operations. These tables are part of the Task Repository functionality of the VMM server engine. It is possible to view an organized report of each task, including its task ID, in the Microsoft SQL Server Management Studio Express application.

To view the Task Trail database, perform the following steps:

1. Open Microsoft SQL Server Management Studio Express.

2. In the left pane, go to: VirtualMachineManagerDB\Tables\dbo.tbl.TR_TaskTrail.

3. Right-click on the dbo.tbl.TR_TaskTrail database and select Open Table.

The TaskTrail database records both user initiated tasks and background scheduled tasks such as refresh operations. The entries in this table are tombstoned at 90 day intervals by default.

Note:It is possible to modify the tombstoning frequency by changing the TaskGC value, defined in days, in the following registry key:

HKEY_LOCAL_MACHINE\Software\Microsoft\Microsoft System Center Virtual Machine Manager Server\Settings\Sql

This information is also contained in a KB:

The Virtual Machine Manager service may consume high memory or CPU utilizationhttp://support.microsoft.com/kb/2009348

Some of the key entries in the Task Trail database include:

Task ID: Guid

Task State: Success/Failed

(Task) Description

34

Any error codes encountered

Start and End date time

PowerShell Commandlet name

Owner: User account which initiated the task

Was the user notified of the task success or failure via a message or error?

After locating the Task ID of the task which failed, there are a number of methods for isolating that specific task within a trace.

Method 1: .CAR file and FINDAfter converting the ETL trace to a CAR file, run the following command at a Command Prompt to pipe all of the lines in the trace relating to the specific Task ID (obtained from the Task Trail database) to the taskid.txt text file:

find /i /n “(Task ID)” path_to_car_file.car >taskid.txt

This method is also very useful when using the PID and TID of a Task.

Method 2: TextAnalysisTool.netAfter converting the ETL trace to a CAR file, open the file in TextAnalysisTool.net.

1. Click on Filter > Add filter.

2. Enter the Task ID obtained from the Task Trail database and click OK.

3. Click View and select Show Only Filtered Lines.

WinRM TroubleshootingWindows Remote Management (WinRM) provides the communication services for communication between the VMM management server and the VMM Host agents. This includes inter agent communications to the following hosts:

Hyper-V Virtual Host Servers

Virtual Server Host Servers

Remote Library Servers

The SCVMM 2008 R2 VMMServer service utilizes the WinRM scripting API (versus COM) for communication with the Windows Remote Management service on the managed hosts.

35

Tip:WinRM error messages usually provide a hex error code that is useful in understanding the problem. To resolve the hex code to error message type:

winrm helpmsg 0xXXXXXXXXIf the hex code is not preceded by ‘0x’ this command will not work.

Test local WinRM functionalityWhen troubleshooting WinRM treat the process as if performing a simple network ping test. Type the command below at an elevated command prompt:

winrm id

This should produce output similar to below:

IdentifyResponse ProtocolVersion = http://schemas.dmtf.org/wbem/wsman/1/wsman.xsd ProductVendor = Microsoft Corporation ProductVersion = OS: 6.1.7201 SP: 0.0 Stack: 2.0

If an error is generated as opposed to the output above, perform a Quick Configuration of WinRM:

winrm qc

You may be prompted with something similar to below. Answer ‘Yes’ to any requests.

WinRM already is set up to receive requests on this machine.WinRM is not set up to allow remote access to this machine for management.The following changes must be made:

Enable the WinRM firewall exception.

Make these changes [y/n]?

WinRM has been updated for remote management.

WinRM firewall exception enabled.

WinRM can now be tested again by typing ‘winrm id’ as before.

Test remote WinRM functionalityThe second half of a WinRM test establishes that the remote server has WinRM configured correctly. At an elevated command prompt type:

winrm id –r:<remoteserver>

This should produce output similar to below:

winrm id -r:vmmr2lab-cl20IdentifyResponse ProtocolVersion = http://schemas.dmtf.org/wbem/wsman/1/wsman.xsd

36

ProductVendor = Microsoft Corporation ProductVersion = OS: 6.1.7201 SP: 0.0 Stack: 2.0

If instead an error such as below appears, this means WinRM is not set up correctly on the remote machine, or there is something preventing communication over port 80 between the two systems. This could be a firewall or antivirus/malware programs.

WSManFault Message = The WinRM client cannot complete the operation within the time specified. Check if the machine name is valid and is reachable over the network and firewall exception for Windows Remote Management service is enabled.

Error number: -2144108250 0x80338126The WinRM client cannot complete the operation within the time specified. Checkif the machine name is valid and is reachable over the network and firewall exception for Windows Remote Management service is enabled.

In this event, test local WinRM functionality on the remote system. If WinRM is configured correctly on the remote system as well the cause is most likely network communication between the two machines. Troubleshoot this as you would any network issue.

WMI TroubleshootingWhen troubleshooting VMM issues that appear to be related to WMI, it is important to note that WMI errors may simply indicate the inability to communicate with the WMI service on a remote machine, not necessarily a problem with the service. This could be due to permissions or network-related issues. It is best to test and troubleshoot suspected WMI issues outside of VMM, using the WMI troubleshooting tools that are available.

WMI troubleshooting tools and related information is discussed in this section.

Note:It is not within the scope of this document to discuss WMI troubleshooting in detail. The information provided in this section is meant to provide the reader with basic knowledge to assist in determining if additional WMI troubleshooting is required.

WMI Troubleshooting ToolsThe following tools may be used to test and troubleshoot the majority of WMI issues:

Windows Management Instrumentation Tester (wbemtest.exe)

WMI Service Control Utility (winmgmt.msc)

WMIC command line

37

More Information:More information about WMI troubleshooting is available at the following location:http://msdn.microsoft.com/en-us/library/aa394603(VS.85).aspx

Windows Management Instrumentation Tester (wbemtest.exe)The Windows Management Instrumentation Tester (wbemtest.exe) provides the ability to connect to and query WMI namespaces, on local or remote machines. It may be used to verify that an object exists in a particular namespace.

More Information:More detailed WBEMTest usage information available at the following location:

Wbemtest.exe is well suited to quickly test connectivity to local and remote WMI namespaces. Connecting to a namespace on a local machine verifies that the namespace is properly registered and accessible via the WMI service. Connecting to a namespace remotely additionally verifies that WMI connectivity between the two machines is working.

Verify WMI NamespacesEach namespace below represents a point of failure that can be tested. To open this tool, type ‘wbemtest.exe’ at the Run prompt. The following screenshot shows the method used to connect to the remote server VMM2008R2-03C. The ‘Connect…’ button has been clicked on the ‘Windows Management Instrumentation Tester’ window, then the name of the remote server appended to the beginning of the ‘root\cimv2’ namespace. Below are three namespaces to verify.

Verify basic WMI service availability by connecting to ‘root\cimv2’

Verify that the virtualization namespace (Hyper-V) is available by connecting to ‘root\virtualization’

Verify that the VMM Agent namespace is available by connecting to ‘root\scvmm’

Note:The SCVMM namespace will not be available unless a Host or P2V Agent is installed and running on the target system.

38

Figure 9: wbemtest.exe used to connect to remote namespace

WMI Service Control Utility (winmgmt.msc)The WMI Service Control Utility configures and controls the WMI service. This tool allows permissions to namespaces to be viewed and modified. To open this tool type ‘winmgmt.msc’ at the Run prompt. In the following screenshot, ‘WMI Control (Local)’ has been right-clicked and ‘Properties’ selected. ‘Root’ was then selected on the ‘Security’ tab and the ‘Security’ button clicked in the lower right of the window. At this point permissions can be viewed and modified.

39

Figure 10: winmgmt.msc Security settings

WMIC command lineThe WMIC command line tool is useful for quickly pulling information about local or remote systems. The same namespaces as described for the ‘wbemtest’ tool can be verified as follows:

LocalTest local CIMV2 namespace.

wmic /namespace:root\cimv2 context

Output below verifies connectivity.

40

NAMESPACE : root\cimv2\root\cimv2ROLE : root\cliNODE(S) : VSTATIONIMPLEVEL : IMPERSONATE[AUTHORITY : N/A]AUTHLEVEL : PKTPRIVACYLOCALE : ms_409PRIVILEGES : ENABLETRACE : OFFRECORD : N/AINTERACTIVE : OFFFAILFAST : OFFOUTPUT : STDOUTAPPEND : STDOUTUSER : N/AAGGREGATE : ON

Other namespaces to check:

wmic /namespace:root\virtualization contextwmic /namespace:root\scvmm context

RemoteTest remote namespaces.

wmic /node:"vmm2008r2-03c" /namespace:root\cimv2 contextwmic /node:"vmm2008r2-03c" /namespace:root\virtualization contextwmic /node:"vmm2008r2-03c" /namespace:root\scvmm context

WMI Error ConstantsThe table below provides information regarding ranges of WMI return codes.

Table 2: Common ranges of WMI errors

Term Description

0x800410xx - 0x800440

Errors that originate in WMI itself.A specific WMI operation failed because of:

• An error in the request, for example, a WQL query fails or the account does not have the correct permissions.

• A WMI infrastructure problem, such as incorrect CIM or DCOM registration.

0x8007xxx Errors originating in the core operating system. WMI may return this type of error because of an external failure, for example, DCOM security failure.

0x80040xxx Errors originating in DCOM. For example, the DCOM configuration for operations to a remote computer may be incorrect.

0x80005xxx Error originating from ADSI (Active Directory Service Interfaces) or LDAP (Lightweight Directory Access Protocol), for example, an Active Directory access failure when using the WMI Active Directory providers.

41

A complete list of WMI errors and their meanings may be found at the following location:http://msdn.microsoft.com/en-us/library/aa394559(VS.85).aspx

BITS Troubleshooting

IntroductionBackground Intelligent Transfer Service (BITS) transfers files (downloads or uploads) between a client and server and provides progress information related to the transfers. You can also download files from a peer. BITS can transfer files asynchronously between a client and a server. VMM uses BITS to transfer payload between managed computers. These data transfers are encrypted by using a self-signed certificate generated at the time a host machine is added to VMM.

Verify BITS outside of VMMThe following steps can be used to verify that BITS is working properly outside of VMM.

BITSadmin can be downloaded here: http://msdn.microsoft.com/en-us/library/aa362813(VS.85).aspx

BITSadmin examples here: http://msdn.microsoft.com/en-us/library/aa362812(VS.85).aspx

42

1. Download BITSadmin.exe using the above link. It is also in the Windows Server 2003 Resource Kit. It does not appear in the tools directory, but is accessible from the command line.

2. Click on … Start > Programs > Windows Resource Kit Tools > Command Shell

3. This should open an elevated command prompt

4. Create a C:\Temp directory if one does not already exist

5. Type the following command: bitsadmin /transfer myDownloadJob /download /priority normal http://msdl.microsoft.com/download/symbols/debuggers/dbg_amd64_6.11.1.404.msi c:\temp\dbg_amd64_6.11.1.404.msi

6. This command will automatically use BITSadmin.exe to test BITS functionality by downloading a file from our website to the local temp directory

7. This should take 10-20 minutes to complete

8. This test should be ran on the VMM management server, Host computer and Source machine

BITS Compact ServerThis method will allow you to test BITS in the same manner in which VMM uses BITS.

On the destination host, try the following steps:

1. Disable BITS Compact Server. This feature is found in the Add/Remove Feature wizard.

2. Restart computer

3. Re-enable BITS Compact Server

If you tried these steps and still have problems, run the following command on the host machine:

winrm invoke CreateJob wmi/root/microsoft/bits/BitsClientJob @{Displayname="Test of fake job";RemoteUrl="http://download.microsoft.com/download/D/0/E/D0E6D2C1-2593-4017-B26D-7375BC9263D5/PowerShell_Setup_amd64.msi";LocalFile="PATH_TO_LOCAL_DESTINATION";Type="0";ServiceAccount="0";Suspend="true";Description="Description for fake job"}

Please replace PATH_TO_LOCAL_DESTINATION with a path to an existing directory, but to a non-existing file (e.g., c:\httpRec\test.txt, where directory c:\httpRec exists, but test.txt in that directory does not). Don't provide path to an existing file, as it might overwrite it. This will attempt to create a BITS client job to download a file from a fake url.

Take note of the output.

If job succeeds, record the JobId.

Example:

CreateJob_OUTPUT    JobId = {8DC2BE2F-0D2A-41B8-AEAD-F6DBED586E98}    ReturnValue = 0

If above command succeeds, clean up the job by the following command:

winrm Invoke SetJobState wmi/root/microsoft/Bits/BitsClientJob?JobID={8DC2BE2F-0D2A-41B8-AEAD-F6DBED586E98} @{JobState="0"}

43

Replacing jobID with the job ID for your job. If the JobId was not returned with the CreateJob command above, you can find created dummy job by:

bitsadmin /list /allusers

Look for job with above name/description.

Also look for any Suspended jobs and delete by using the following commands:

Bitsadmin /cancel {Job_GUID}

BITS TracesOccasionally it will be necessary to obtain BITS traces while reproducing the error.

Create bitslog.cmd using below batch fileOpen an elevated powershell and navigate to the "C:\bits folder"Type the following commands:     Bitslog /enable     Bitslog /collect c:\bits                     # choose an appropriate directory namePerform operations and reproduce failureType the following commands from powershell     Bitslog /disableCollect the bits.log file for analysis

44

The following batch file should be renamed bitslog.cmd

@echo offREM Script for enabling bits logging/collection to be usedREM while reporting BITS issues

setlocal ENABLEDELAYEDEXPANSION

set DEFAULT_LOG_SIZE=20set DEFAULT_LOG_DIR=%TEMP%\bits-logsset BitsKey=HKLM\Software\Microsoft\Windows\CurrentVersion\BITS

if {%1} == {} goto Usage

if {%1} == {/enable} goto :EnableLogif {%1} == {/e} goto :EnableLog

if {%1} == {/disable} goto :DisableLogif {%1} == {/d} goto :DisableLog

if {%1} == {/collect} goto :CollectLogif {%1} == {/c} goto :CollectLog

45

goto :Usage

:EnableLogif {%2} == {} (set log_size=%DEFAULT_LOG_SIZE%) else (set log_size=%2)echo Enabling logging for BITS with log file size as %log_size%reg add %BitsKey% /v LogFileFlags /t REG_DWORD /d 0xfbcf /f > NULreg add %BitsKey% /v LogFileSize /t REG_DWORD /d %log_size% /f > NUL

echo Restarting BITS for registry values to take effectnet stop bits net start bitsgoto :eof

:DisableLogecho Disabling logging for BITSreg delete %BitsKey% /v LogFileFlags /f > NULreg add %BitsKey% /v LogFileSize /f > NUL

echo Restarting BITS for registry values to take effectnet stop bits net start bitsgoto :eof

:CollectLogif {%2} == {} (set log_dir=%DEFAULT_LOG_DIR%) else (set log_dir=%2)if NOT EXIST %log_dir% (md %log_dir% > NULif ERRORLEVEL 1 (echo Failed to create the log dir %log_dir%. Not saving logsgoto :eof))echo Copying the BITS logs to %log_dir% directoryREM Flush the current log for BITSlogman update bits -ets -fd > NULcopy %windir%\system32\bits.log %log_dir% /ycopy %windir%\system32\bits.bak %log_dir% /y

REM copy the build inforeg query "HKLM\Software\Microsoft\Windows NT\CurrentVersion" /v BuildLab > %log_dir%\bld_info

REM get the output of bitsadminbitsadmin /list /allusers /verbose > %log_dir%\AllJobs.txt

46

goto :eof

:Usageecho BITSLOG usageecho bitslog /enable [^<no^>] - To enable BITS logging and specify the log size as ^<no^> MBecho BITS service will be restartedecho If ^<no^> is not specified, default is 20 MBecho bitslog /disable - To disable BITS loggingecho BITS service will be restartedecho bitslog /collect [^<dir^>] - To collect BITS logs in the specified direcho If ^<dir^> is not specified, default is %%TEMP%%\bits-logsgoto :eof

Useful KB Articles and BlogsHow to Troubleshoot Slow BITS Performance, Hosts 'Not Responding' and 'Needs Attention' Communication Issueshttp://blogs.technet.com/b/jonjor/archive/2008/12/29/how-to-troubleshoot-slow-bits-performance-hosts-not-responding-and-needs-attention-communication-issues.aspx

Transfers between the System Center Virtual Machine Manager server, the Library Server and the Virtualization Hosts may fail with Error 12700 or 2912http://support.microsoft.com/default.aspx?scid=kb;EN-US;2405062

P2V fails with Error 2912 0x80072F0C with System Center Virtual Machine Manager 2008 or System Center Virtual Machine Manager 2008 R2http://support.microsoft.com/kb/2385280

47