Case Studies , CSEIT

Citrix Critical Components Case Studies

“Under The Hood”

Citrix Global Escalation teamCitrix Systems, Inc.

2 Citrix Company Confidential © 2004 Citrix Systems, Inc.—All rights reserved.

Agenda

• Server Critical– IMA

• Installation Troubleshooting• Utilities Programs• Datastore Best Practices

• Session Critical– Session Components

• Client Critical– Client Components

• Open Forum


Presentation CD Contents

• Today’s PowerPoint Presentation

• Case Studies

• IMA Files and Registry Reference Document

• Client to Server Content Redirection Document

• Multi-Monitor Support Document


Server Critical Components

• Independent Management Architecture – Overview

– Troubleshooting Installation

– Troubleshooting Startup Issues

– Improvements

– Data Store Tuning


What is the IMA Service?

• IMA Service –– The main Citrix Service. The IMA service interacts with

the Citrix XML service and Terminal Services.

– The Data Collector is one of the MF XP servers that holds dynamic farm data (IMA)

– IMA maintains a persistent connection to the SQL Server (the Data Store) where it stores static farm data.


IMA Components Review

Local HostCache

Data Store

DynamicStore

Volatile Data

MPS XP Server

Data Collector

IMA TCP Port 2512

MPS XP Server

ODBC

ODBC

Local HostCache

Citrix Management Console

QueryDSWill dump Out Raw

Data

DSCheckDetect and correct Data Store Errors


Troubleshooting with QueryDSCase Study

• Case Study– Users unable to connect to published applications– Qfarm /load shows a zero load value

• Troubleshooting

queryds /table:ServiceTableName: 0

Host: XPSERVER

• Resolution – Chfarm into an access datastore then rejoin the

farm– Review CTX103017 for more information

Should be a HEX number


Starting the IMA Service

Service Control Manager

IMASrv.exeRequired Plug-ins

Product Plug-ins

PSRequiredHKLM\Software\Citrix\IMA\

Runtime

PSRequired=1PSRequired=0

Zone Data Collector

LHCLHC

ImaGrpSs.dll

ImaRelSs.dll

ImaAdminSs.dll

ImaRaSs.dll

ImaRpcSs.dll

ImaSrvSs.dll

ImaAppSs.dll

MfSrvSs.dll

MfBrowserSs.dll

ImaUserSs.dll

ImaDomain.dll

RMAlertsSS.dll

RMMonitorSS.dll

RMSummaryDBSS.dll………………..

ImaRuntimeSs.dll

ImaRuntimeSs.dll


Troubleshooting General IMA Startups Issues

• Review Event Logs

• Verify Registry Keys– HKLM\SOFTWARE\Citrix\IMA\RUNTIME

• CurrentlyLoadingPlugin Indicates which DLL module is being loaded

• PSRequired – If set to one the Datastore must be available – SQL Profiler Audit Logins

• Use DSMAINT– Config – Resets account and DSN to connect to the Datastore

– RecreateLHC – Rebuilds Local Host Cache

IMA Must be able to connect to the Data Store in order for qfarm

to enumerate server

Please review IMAFileandReg for more details


IMA Service Fails to start during installation of MPS 3.0

• Case Study• Installation of XP fails due to IMA service fails to start

• Troubleshooting– Turn on MSI tracing (msiexec /i mps30.msi /l*v c:\msi.log)

– Start IMA service returned error in CTX_MF_IMA_StartIMAService Error = 1068

– net helpmsg 1068

– The dependency service or group failed to start

– Did all of the dependent services start?

• Resolution• Enable The IPSec Services

FR3 Dependencies


IMA Fails to start with File not found Errors or Unable to Locate DLL

Case Study

IMA Fails to start with File no found or Unable to Locate DLL

ResolutionCorrect System PathExample %systemdrive%\Program Files\Citrix\System32\Citrix\IMA


IMA fails to start immediately

Case Study

IMA Fails to start with IMA_RESULT_FAILURE

Troubleshooting

File Monitor Trace

7930 3:56:28 PM ImaSrv.exe:3980QUERY INFORMATION C:\WINNT\TEMPFILE NOT FOUND Attributes: Error

Resolution Verify TEMP directory exists under %SystemRoot%\Temp

and the system account has full control.


IMA Fails after installing Third Party Software

Case StudyIMA Result_Plugin_Failed_Verify in System Event Log

TroubleshootingFile Mon (www.sysinternals.com) trace indicated an invalid path was being

referenced.

ResolutionCorrect the following registry keys values with the correct file path

HKLM\SOFTWARE\Microsoft\Windows\CurrentVersion\Explorer\User Shell Folders - Common AppData

HKEY_USERS\.DEFAULT\Software\Microsoft\Windows\CurrentVersion\Explorer\User Shell Folders


IMA Failing to Start or Long Delays

• Troubleshooting– SQLDIAG

• %systemdrive%\Program Files\Microsoft SQL Server\MSSQL\Binn• Collects SQL Error Logs• Collects SQL Server Configuration• Collects SQL Database Information

2004-1-24 08:46:32.74 spid355 BobMgr::GetBuf: Bob write not complete after 60 seconds

• ResolutionIncrease the size of Tempdb

Stop and periodically stop

and start SQL to purge Tempdb

Avoid the Auto grow operation and

Do not set the Auto Shrink option

TempDB


IMA Change Notification Process

Data Collector

Data CollectorData Store

Citrix Management Console

Member Server

Member Server

Member Server

Member Server

Member Server

Member Server

Member Server

1) Change is made on the CMC via TCP port 2513.

2) The member server writes the change to the DS and forwards the information to the DC via TCP port 2512.

3) The DC updates the LHC on the member servers in its zones via TCP port 2512 and forwards the information to all the other DC’s.

4) The other DCs send the information to their member servers.

25132513

25122512

25122512

25122512

25122512

25122512

Zone A Zone B


Change Polling

• Used as a backup method of discovery

– Example inconsistent WAN Links

• FR3+ 30 minute interval

• Configurable in registry under HKLM-Software-Citrix-IMA-DCNChangePollingInterval

Data Store

Polling SQL statement

SELECT COUNT(*),AVG(imalock) FROM KEYTABLE


Black Hole Syndrome

• What is the black hole syndrome?

• What are the variables that can cause condition?

• How can it be prevented or lessen the impact?

Blackhole - Artistic conception from NASA Hole


WI Logon Process

WI

Server Farm

1. Browser to logon page through WI

2. Using WI’s classes to fwd the information to the Citrix XML Service on servers in the server farms. The designated server acts as a broker between the Web server and servers.

3. The Citrix XML Service on the designated server then retrieves a list of applications from the servers that users can access.

4. The user initiates the next step by clicking one of the hyperlinks on the WI page.

5. The Citrix XML Service is contacted to locate the server in the farm that is the least busy. The XML Service requests a ticket from the least busy server corresponding to the user’s credentials.

6. The classes finish parsing the template file and send a customized file to the Web browser.Client

7. The Web browser receives the file and passes it to the client device.

8. The client receives the file and initiates a client session with a server according to the file’s connection information.


PN Agent Logon Process

WI/Config.xml

Server Farm

1. PN Agent passes a request to the WI web server using XML and Unicode

2. Configuration file on the WI web server is accessed. inetpub\wwwroot\citrix\pnagent directory3. Application enumeration request is passed with user credentials from the WI web server to the MetaFrame XP server using Citrix XML Service.

4. Application set data is generated on the MetaFrame XP server and returned to the WI web server.

5. The WI web server passes the application set data back to the client device, where the application and desktop icons are populated.

6. The user clicks to launch an application.

PN Agent Client

7. The request for an .ICA file is sent to the WI web server.

8. The WI web server uses the TEMPLATE.ICA file located on the web server to generate an .ICA file.

9. The WI web server sends the .ICA file to the client device.


PN Logon Process

Server Farm

1. user clicks the Citrix Program Neighborhood icon, which calls the ICA Win32 Citrix Program Neighborhood Client (PN.EXE).

2. The ICA Win32 Citrix Program Neighborhood Client reads the parameters stored in the .INI files upon execution.

3. PN.EXE sends the request for Application Sets to WCFRUN32.EXE.

4. WFCRUN32.EXE passes the request to WFICA32.EXE, creating a partial session on the MetaFrame XP server. This connection is data only and does not include graphical information.

5. WFICA32.EXE sends the request to the MetaFrame XP server.

6. The MetaFrame XP server generates the data for the application sets and sends the data back to WFICA32.EXE.

PN Client 7. WFICA32.EXE receives the data and sends the information to PN.EXE.

8. PN.EXE creates the application sets in the Citrix Program Neighborhood interface.

9. The user double-clicks an application set.

Data Collector

10. PN.EXE sends the request to WFCRUN32.EXE.

11. WFCRUN32.EXE sends the request to WFICA32.EXE. WFICA32.EXE sends the request to the server.


Troubleshooting with XML/Ticketing Case Study

Launch an Application (get the least busy server )Launch an Application (get the least busy server )

MetaFrame XML Presentation

Server

IMA (local DC)

XML Service

IMA (remote DC)

* DC = Data Collector

XML request Ticket request with credentials

From the Web Interface

• XML asks IMA to provide the IP Address of the least busy server in the farm that has the requested publish application published on it.• If the XP server is not the Data Collector it needs to contact the zone Data Collector for this information


Troubleshooting with XML/Ticketing Case Study

The Least Busy MetaFrame Farm

Presentation Server

Termsrv

wsxica.dll

XML Service

CCTicket.dll

Launch an Application (provide a ticket )Launch an Application (provide a ticket )

credentialsticket

credentialsticketcredentialsticket

CCTicket.dll

XML Service

CCTicket.dll

XML Server

• The XML Server makes an XML request with credentials for a ticket from the least busy server

• This is done via a TCP Socket Connection between the XML services.


Troubleshooting IMA Hang Issues

• IMA Service

Term Serv

Data Store ODBC

Troubleshooting External Dependencies

XE103W2K3022

XE103W2K119

Qfarm

Qfarm /Online

Stop The IMA serivice

Userdump

Kill (Kill –f) IMA PID

• Check MDAC Levels on XP and DataStore

• (SQL 2000 SP 3a)

• Data Store Locks

• (SQL SP_WHO)

• Qwinsta Hangs ? (Qwinsta /debug)

• Quser Hang?

If SP3 See MS 813790


IMA Hang on SQL

• Run the following under query analyzer– Select * from master.dbo.sysprocesses where

program_name=‘Citrix IMA’

• MS KB 884554

• FIX: A SPID stops responding with a NETWORKIO (0x800) waittype in SQL Server Enterprise Manager when SQL Server tries to process a fragmented TDS network packet


IMA Query Timeout

IMASQL.DLL

RM

LM

IM

Print SS

IMASQL Serializes DS access

RM

LM

IM

Print SS

RM

LM

IM

Print SS Database causes threads to queue until IMA is unresponsive

The Query timeout fix will allow IMASQL to close the connect and allow the next thread to be processed


IMA Query Timeout Fix

• XE103W2K136 MPSE300R03W2K008

HKLM\Software\Citrix\IMAName: DB_TIMEOUT_DELAY Type: REG_DWORDData: <timeout delay, in seconds>

HKLM\Software\Citrix\IMAName: DB_TIMEOUT_DELAY Type: REG_DWORDData: <timeout delay, in seconds>

HKLM\Software\Citrix\IMAName: DB_CONNECTION_TIMEOUT Type: REG_DWORDData: <timeout value, seconds>

HKLM\Software\Citrix\IMAName: DB_CONNECTION_TIMEOUT Type: REG_DWORDData: <timeout value, seconds>


Database corruption issues

DSVerify 1.0

1. Always backup IMA database before running

2. DSCheck is the only utility needed in the future

3. Backup DS before running

4. Run with /full option to see all advance settings


IMA Best Practice

Adjust Polling IMA Polling Interval DCNChangePollingInterval

Set Data Collectors to latest version

Use Data Store Replication to reduce traffic and increase performance

Oracle Data Store CTX103685 Use Oracle Client 8.1.7.2


Data Store SQL Best Practices

Service Pack 3a or 4 for SQL 2000

Maintenance Plan for SQL

Stop and restart SQL to flush TempDB on a scheduled basis in large farms

Set XP DS and TempDB to large size to limit auto grow

Make sure Auto Stats Option is turned on

Backup and purge transaction logs or change recovery model from Full to Simple

Data Store


Session Critical

• Connection Sequence• Session Processes• Session Sharing


Session Initialization – Flow Chart

Client Makes an

Application Request

ICA ListnerStack

Receives theRequest On 1494

Session stackNotifies ofSuccessfulconnection

Session stack isMapped to an

IdleSession

The TerminalServices

Device driverForwards the request

To TermSRV

New SessionStack is created

Listner stack handsThe connection

Request to the new Session stack

Session connectionDetails areNegotiated

Requested application

Is launched in theAppropriate session

space

New domainUser & SessionID

Mapping are created

Credentials areAuthenticated & SessionID

is Assigned orReloaded

Windows LogonPrompts for User

Credentials


Virtual Channels

ICA Hand Shake 7F7FICA

Authenication Phase

Authenication

Virtual Channels

Client Capability Phase

TS CAL Check

TS CAL Verification PhaseConnection Initialization Phase

Session Initialization Overview

`

TCP Port 1494

ICA Hand Shake 7F7FICA

Module.INI

Metaframe Server

ICA Client

CTX564283 How to Troubleshoot 1003 and 1004 Event Log Errors Regarding Terminal Server Licensing


TSCal Explained

Session Connection

Begins.

Is their a valid Token

Present?(Windows 2000 or

XP)

Terminal ServerAttempts to connect

to TS Cal server

License server

Found?

Discovery process startsto find a License Server

Connection is denied unless

A Temp. Token is present

Connection is Allowed

AnotherTS Cal server

w/ TSCalsFound?

TSCals avail.Available on

TS Cal Server?

License server responds

To the Terminal serverAnd a TS Cal is

requested

Session denied orTemp Token is

Issued andSession is allowed.

Original TS Cal server Redirects

the TerminalServer to new

License Server.

Token is markedOn TS Cal

Server. Cert. is sent Through the Terminal

Server to Client.


Winlogon.exe

Userinit.exe

Cmstart.exe

Wfshell.exe

Session Initialization

Session Process Overview


Wfshell Overview

• Wfshell runs in every ICA session– Launching Published Applications

– Auto-Creating Printers

– Executing the RunOnce REG Key

– Session Virtual Channels • HKLM >SYSTEM>CCS >Control >Citrix

>Wfshell• Speed Screen / SSBA• Clipboard Mapping• Shadow Indicator• Session Sharing• Content Redirection• TZ Redirection

TroubleShooting – LISTDLL (www.sysinternals.com) Will list all dependant DLL’s and paths where they were loaded from.


Slash of Death Issue “/”

Case Study Session Sharing is not occurring in user’s sessions.

Printing, Clipboard mapping, Content Redirection Failures…etc.

Troubleshooting Verify if Wfshell.exe running in the user’s session (CMC)

Check Dr Watson Logs

Resolution In one case wfshell.exe “crashes” were occurring in the user’s

session. This was caused by older HP drivers and the new client printer name convention in MPS Windows 2003.

This was fixed by using the latest drivers from HP’s Web Site.

Client/FTLRENEAIBMPAD#/HP Laser Jet 5000 Series PCL 6


Virtual Channel Case Study

Wfshell functionality not working for domain users

RDP having similar issue with RDPCLIP

RUNAS under admin account works

Process Explorer from www.sysinternals.com

\Device\Termdd permissions object

Resolution

Remove or update the userdump utility program


Client Critical

• Win32 ICA Client Core Components

• Client Case Studies

• What’s New in ICA Win32 8.0 and 9.0 Clients ?

• Support News


WFICA32Client Engine

WFCRUN32

WFICA.OCX

Supporting Files

Configuration Files

PN - PNAgent - Web Client

Win32 Client Family

Win32 ICA Client Components

Client EngineHosts Session

Maintains SessionsConnection Center

[Systray Icon]WI MSAM ICOExposed Public API

Functional DLL’s

Client INI Settings


Win32 ICA Client – Core Components

When a Published application is launched, this file is responsible for the connection to the server for the duration of the session.

Wfica32

Client Engine

wfica32.exe

• Client Seamless Engine (8.0 and earlier, 8.1 and later has a new file, VDTWIN.DLL )

• Client Clipboard Engine

• Hotkey Functionality.

• Loading additional Client DLL’s for increased functionality.



Wfcrun32.exe

Client connection manager and is launched by PN, PNA and wfica32 (if a

seamless connection).

It also manages:

The security dialogs. The client queries security settings and wfcrun32 may display

the security dialog.

You cannot session share if wfcrun32 was not launched.

Wfcrun32



Wfica.ocx

Used in Web Interface, MCM, MSAM and embedded web ICA sessions.

Provides equivalent functionality as Wfcrun32 in the form of an Active X

control.

Exposes the ICA Client Object Interface (ICO)

For more information on the ICO, see the following Online PDF file: CTX103768

WFica.ocx



There are several supporting files that provide additional functionality to the Core Client Engine

Supporting

Files

Virtual Drivers

Protocol Drivers

Network Drivers

Transport Drivers

Additional Files.


Win32 ICA Client – Configuration Files - PN

Appsrv.ini Module.iniPN.iniUIstate.ini Wfclient.ini

Hot KeysConnection Settings

Wfica Engine SettingsClient CapabilitiesDriver SettingsPN App Set DataPN UI SettingsWindow Panning and

ScalingWindow Sizing and

Position Kerberos SSON

SettingsKeyboard Layout

Client INI Files

Config

Files


Win32 ICA Client – Configuration Files - PNA

PNAgent Specific:- Retrieves its “Ini” settings from the Config.xml stored on the W.I. server. - ICA files from Template.ica.- Other settings are in the local registry.

`

config.xml

Web InterfacePNAgent

Enum.asp, Launch.asp

Tip: PNAgent Caches .ica files in the directory:

%UserProfile%\Application Data\Citrix\PNAgent\ResourceCache


Pass Through Client Settings

`

Client Name B

Server Name

`

Client Name A

Server Name

Server1Server2

Client Name A

Client Name B

Pass Through ServerDestination Server

HKLM\SOFTWARE\Citrix\ICA Client\PASS THROUGHPASS THROUGH REG_DWORD = 1

HKLM\SOFTWARE\Citrix\ICA Client\PASS THROUGHPASS THROUGH REG_DWORD = 1

Clients


Pass Through Client Settings

HKLM\Software\Citrix\Program Neighborhood Agent

EnablePassThroughFTA

REG_DWORD =1

HKLM\Software\Citrix\Program Neighborhood Agent

EnablePassThroughFTA

REG_DWORD =1

Appsrv.ini

ForceLVBMode=1DeferredUpdateMode=On

Appsrv.ini

ForceLVBMode=1DeferredUpdateMode=On

Enable Client to Server Content Redirection in a PassThrough Session

Screen Refresh Issues Through a Seamless Pass-Through Session - CTX204071

Enable Server Drives to Appear as Client Drives when using the PassThrough Client - CTX238200

module.ini

[client drive]

NativeDriveMapping=TRUE

module.ini

[client drive]

NativeDriveMapping=TRUE


Client Side - Session Sharing Troubleshooting

Connection Center

Present in PN, Full Web Clients and now PNA

(8.0 or later).

Allows users to manage connections and

Security Dialogs.


Client Side - Session Sharing Troubleshooting

`

Wfshell.exe

Word Application

Response

Excel Application

Response

Default Timer = 20 Seconds

Launch in Session

Win 32 Client

Appsrv.iniSucConTimeOut

=40

Word

Excel

Network Request

Queued Virtual Channel Request


• Problem

Users behind a proxy server are disconnected if the proxy server is set to disable connections based on some specified timeout value.

Case Study : Disconnected sessions through a proxy server.



Proxy Server

FirewallWI 3.0 Server

MPS 3.0 Server

XML

ICA/Http(s)

ICA/Http(s)

ICA/Http(s)

ICA

Http/s


Troubleshooting Procedures

Enabling ICA Keepalives on the MPS3.0 Server

The exact path to those value(s) are as follows Key: HKLM\System\CurrentControlSet\Control\CitrixValue :ICAEnableKeepAlive = 1 :ICAKeepAliveInterval = (Delta < Proxy_Idle_Time_Value)


Some proxy servers may be intelligent enough to terminate idle sessions if the KeepAlive packet is not from the initiator.


• The ICA Client (starting with the current release) has the ability to send a KeepAlive packet to the MPS3.0 server.

• This prevents ICA sessions from being disconnected if the proxy device is configured with an Idle time disconnect from the session initiator.


Proxy Server

ICA


AppSrv.iniUsers Profile

`

Case Study : Disconnected sessions

through a proxy server.

Template.icaOn Web Interface

ICAFile.xsltOn MSAM


New Features Starting with 9.0

Multi-Monitor Improvements(CPS 4.0)

TWAIN support(CPS 4.0)

PNA Roaming SmartCard(CPS 4.0)

Advanced Universal Printing(CPS 4.0)

PDA Synchronization(ActiveSync) (CPS 4.0)

Client-side File Locking(CPS 4.0) Pass-Through Windows Key support

Program Neighborhood Quick Launch Bar


New Features Starting with 8.0

Session Reliability(requires MPS 3.0 or later)

Logon Look and Feel Dialogs

Kerberos SSON Support(requires MPS 3.0/W.I. 3.0 or later)

PNA support for Multiple Farms (requires MPS 3.0/W.I. 3.0 or later)

Workspace Control (requires MPS 3.0 & WI 3.0)

Bi-directional Audio (requires MPS 3.0 or later)

Video Playback(requires MPS 3.0 or later)


Server to Client Content Redirection

Ability to specify which Content Types are

redirected:

http, https, rtsp, rtspu, pnm, mms

Ability to specify which base URLs can be redirected:

Eg. *.citrix.com

See SCCR.doc on your class CD.

`

Recent Enhancements


Support News

- 9.0 Client Updates List:http://support.citrix.com/kb/entry.jspa?externalID=ctx105677#P20_2735

- Updated Client INI Configuration Guide

- New ACM Client Package

- HRP01 for MF XP 1.0 SP4

- Service Pack 2005.4 for MPS 3.0http://support.citrix.com/kb/entry.jspa?

entryID=6211&categoryID=619#P51_8121

- 9.100 Hotfix Client Information


Open Forum


Session Review

Server Components

IMA Service

Data Store

ICA Session

Connection Sequence

Session Processes

ICA Client

Configuration and Tuning


Thank You


Case Studies , CSEIT

Documents