CCAT Troubleshooting Training – Provisioning Services April 2012 Citrix Consulting Architecture Team.

CCAT Troubleshooting Training – Provisioning Services

April 2012

Citrix Consulting Architecture Team

• PVS Troubleshooting – Key Areas

• WireShark Analysis

• System Memory Dumps

• Key Takeaways

Agenda

Troubleshooting PVS Issues

PVS Ports used - RecapService Protocol Port Comments

DHCP UDP 67 DHCP Server port

  UDP 68 DHCP Client port

PXE UDP 67 If PXE separate from DHCP server

  UDP 4011 If PXE on DHCP server

TFTP UDP 69 Trivial File Transfer Protocol

TSB UDP 6969 Two-Stage Boot Service

Console TCP 54321-54322 SOAP Service

Database TCP 1433 Microsoft SQL Server

Stream Service UDP 6890-6909 Inter-server communications

  UDP 6910-6930 Streaming, initial login on port 6910

Troubleshooting

• Narrowing down scope of issue by determining where in the boot cycle the issue is occurring.

• Packet Traces• Capturing Kernel Dump Files

Boot Stages

• DHCP – PXE boot downloading of bootstrap file• Login to the PVS Server• Streaming of the I/O Pre-BNIStack (Citrix Target Device

driver)• BNIStack up burst traffic

Using Wireshark – Packet Trace Analysis

Analyzing Packet Captures

Packet Capture PXE TFTP download

TFTP

Provisioning Server 10.1.1.1

DHCP

Target 10.1.1.100PXE

Common Issues During DHCP TFTP Download

Target does not receive IP address

Target gets IP but times out to TFTP

PXE Errors No boostrap found

TFTP

Provisioning Server 10.1.1.1

DHCP

Target 10.1.1.100PXE

Login Process Traffic

TFTP

Provisioning Server 10.1.1.1

Target 10.1.1.100

Stream Process

TFTP - Login

Common issues during login process

Incorrect Bootstrap IP information

Login port blocked

Target not entered in Database

Stream Service failed to start

TFTP

Provisioning Server 10.1.1.1

DHCP

Target 10.1.1.100PXE

Streaming Pre-BniStack

TFTP

Stream Process

OS Booting UNDI Driver

Target 10.1.1.100

Common issues pre BNIStack

• Large Send Offload

• Network instability

• Conflicts with UNDI Driver

Determining if BNIStack Driver is up

TFTP

Stream Process

OS Booting BNISTACK Driver

Target 10.1.1.100

Provisioning Server 10.1.1.1

Common issues post BNIStack

• Filter Driver conflicts

• Firewall/Antivirus Services kickoff

• Antivirus Updates being loaded

• Domain Profile creation

Capturing Windows System Dumps

Windows Kernel Dump files

Capturing Kernel Dump Files

• Capturing Kernel Dump files using Win2k3 and XP

• Capturing Kernel Dump files using Win2k8 Vista and Windows 7

• Capturing Kernel Dump files in XenServer and ESX

Capturing Kernel Dump Files Win2k3 and XP Limitations

• Crash dumps to be on same drive as pagefile

• Page file to be on the system drive

• Redirection of dump not supported

• Live debug can be accomplished via serial cable.

Capturing Kernel Dump Files Win2k8 Vista and Windows 7

• Enhancements allow kernel dumps to be directed to a location other then a system drive

• Requires a local hard drive with enough free space for the dump file. (Pagefile Size)

Capturing Kernel Dump Files Win2k8 Vista and Windows 7

• Create a new string value under HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Control\CrashControl named DedicatedDumpFile

• Create Value data box, type <drive>:\<dedicateddumpfile.sys>

• Create DWORD Value DumpFileSize use decimal under base and type the name of dump file in megabytes

• More detailed information in this process can be found:http://support.microsoft.com/kb/969028 http://support.citrix.com/article/ctx123642

Capturing a Kernel dump via XenServer

• Vista, Windows 7, Win2k8 redirection only possible if XenTools are not installed

• Live Debug via WinDBG possible by redirecting serial to DOM 0

Capturing a Kernel Dump Via Xenserver

DO

M 0

XEN SERVER

WinDBGSockPipe.EXESerial Redirect

http://support.citrix.com/article/ctx124979

Capturing Kernel dump via ESX

• Workstation 7 utility vmss2core

• Suspend hung VM

• Download suspended VM session (.vmss)

• Retrieve downloaded file and run command vmss2core –W filename.vmss

http://support.citrix.com/article/ctx125086

Booting Target Devices – TFTP vs. BDM

• Whiteboard Discussion

PVS Retries – Potential Causes

• Network Traffic • High bandwidth utilization• Congested pipe • Application layer filtering• Physical network problems• UDP (LSO, Checksum Offload) Issues

• Check the vDisk Properties (System Tray)

• How to fix these issues? (Discussion)

Key Takeaways

Key Takeaways

• Stream Process Workflow

• How to identify where in the boot process a PVS Target is failing

• Steps to capture a kernel level dump file from a provisioned target