AOS Troubleshooting Tips March 25 2010

AX AOS TROUBLESHOOTING TIPS

AOS Troubleshooting Tips and Techniques

Agenda

AOS ConfigurationBest PracticesTroubleshooting AOS issuesFinding Sessions and SPIDSAOS Crash MaintenanceTools for Troubleshooting AOS issues

AOS Configuration

Application Object Server ConfigurationDatabase Tuning TabTracing TabPerformance Tab (Dynamics AX 2009)

AOS Configuration

Dynamics AX Server Configuration Utility

Configuration stored in Registry:HKLM\SYSTEM\CurrentControlSet\Dynamics Server\4.0\instance\config name

Multiple AOS instances and configurations

Server Configurations

Each AOS instance listedEach AOS has configurations

Application Object Server Tab

Configuration command to run at startupAllowing clients to print from AOS printersEnabling breakpointsFatal error reportingLoad Balancing

AOS Load BalancingNumber of client sessionsWhat is a load balancer How connections work when using load balancingFactors when deciding if you need a load balancer

AOS Load Balancing

Provides fail over abilityBalances number of clients between AOS’Control the number of clients per AOS

AOS without clustering

With no options marked in the load balancing section each client will connect to the AOS in their client configuration

Each AOS balancing load

As a client connects to an AOS that AOS queries the database to determine which AOS the client will connect to

AOS as Load Balancer

The load balancer makes the connection to the database and determines which AOS the client will connect toThis AOS will not accept any client connections nor do any processingThis AOS does not count against your AOS licenses

AOS Load Balancer - Client Configuration

In the Dynamics AX 4.0 Configuration utility you will want to specify the machine name and port number for the AOS load balancer instanceAdd AOS member instances after the load balancer

Database Tuning Tab

Max open cursorsMax buffer sizeAllow index hints in queries

Tracing Tab – Server Settings

RPC round trips to serverX++ method callsFunction callsStart and Stop Tracing

Tracing Tab – SQL Settings

SQL statementsRow FetchConnect and disconnectTransactions: TTSBegin,TTSCommit,TTSAbortBind variablesRow fetch summary

Dynamics AX 2009 featuresPerformance tab

64 bitBatchingTime Zones

Best Practices

Separate AOS’Separation of environmentsMonitoring performanceCode deploymentApplying latest Kernel hotfixes

Separate AOS’

Consider separating out the following processes / components based on load

Additional Dynamics AX componentsThird PartiesBatch

Separate out environments

Servers for production environment should be on dedicated boxes

Server resources dedicated to production performanceReduce risk of development code utilizing majority of Servers resourcesApplication of hotfixesAdditional components that need .Net BC

AOS Baselines

Performance MonitorMemory CPU

Memory usageDifferent times of dayWhen specific processes run

Proper code deployment

Code promotion schedulePromote code during down timeDetermine if you will use xpo’s or layers to move code from development/test to productionTest code in environment that matches production before deploying

Apply Kernel Hotfixes

Improved StabilityApply on all machines with a Dynamics AX components

Dynamics AX clientEnterprise PortalSSRS machineAny machine with .Net Business Connector

Troubleshooting AOS errors

Application event logRPC errorsConnection issuesAOS Crashing

Application event log

Windows default event logging toolDifferent types of events

AOS startingUser errorsCache flushingCrashes

Events related to connection

Lost connection to AOSCannot or lost connection to SQLLost connection to the AOS

Connection to AOS

Object Server 01: RPC error: Client provided an invalid session ID 75

This error message indicates that the AOS has crashed. After the AOS crashed a client that was open when the crash occurred tried to connect to the AOS. Since the AOS was restarted it does not recognize the existing client connection. To resolve this issue the client has to be shut down and restarted.

Connection to AOS

Object Server 01: No ping from 77. Terminating the session.

This error message indicates that the end user ended the client session before the AOS could return the requested results to the client. To prevent this error message end users shouldn’t end task (ctrl –alt – delete) their Dynamics AX session. Instead an end user should contact their AX administrator and have them end their session from the online users form.

Connection to AOS

Object Server 01: SPID 57 for Session Id 77, is still present in the database. Please delete the SPID from the database.

This indicates that the end user shut down their Dynamics AX client before SQL finished processing the request.You can delete the SPID or let SQL finish processingIf you don’t clean up the SPID it could cause issues such as blocking transactions, and open transactions which may lead to transaction log backup failures.

Connection to SQL

Object Server 03: Fatal SQL condition during login. Error message: "[Microsoft][ODBC SQL Server Driver][SQL Server]Login failed for user 'NORTHAMERICA\MBSAXAPTA04$'.“

This typically indicates the AOS service account does not have permissions to the database.

*Typically followed by error on next slide

Object Server 03: SQL diagnostics: [Microsoft][ODBC SQL Server Driver][SQL Server]Login failed for user 'NORTHAMERICA\MBSAXAPTA04$'.. Connect information was: Userid = [], Database = [DAX4_SP1_Prod], Server = [server1], DSN = [], Other = []

This is caused by insufficient SQL permissions. To resolve this issue please use Knowledge base article 911380 method 2.

Connection to SQL

Object Server 03: Internal Error occurred executing stored procedure when creating session for the AOS.

This indicates the AOS Service account does not have adequate permissions to the stored procedures in the database.To resolve this issue please use Knowledge base article 911380 method 2..

Connection to application files

Error in file: C:\Program Files \DynamicsAX\ Application\appl\prod \axsys.aod while reading in record 215B1D0Windows Error: = Error Code: 64 = The specified network name is no longer available.Try again?

This error indicates the AOS has lost connection to the Application files.To resolve this error message you need to restart the AOS.

Finding Sessions and SPIDS

Find a user using a session ID in SQLFind a SPID using a session ID in SQLFinding User and computer from the AOT

Find User and Computer in SQL referenced in the Event Log

If the following event was logged on 6/1/2009 you could run the query below. Object Server 01: No ping from 77. Terminating the session.

select * from sysuserlog where sessionid = 77 and createddate > '2009/06/01' order by createddate desc

How to find an orphaned spid and associate it with an AX session ID

1. Check the status of the Microsoft Dynamics AX sessions, verify that these sessions have active SPIDs in the Microsoft Dynamics AX database where the client session is in the shut down process. To do this, run the following SQL statement:

Select * from SYSCLIENTSESSIONS where status = 2 or status = 3In the results of the statement, check the matching session ID from the warning message that appears in the Application log on the AOS server

2. Check the status of the orphaned SQL SPID. To do this, run the following SQL statement:

Select * from master..sysprocesses In the results of the statement, locate the SPID that is indicated in the warning message, and then verify the following:

If the SPID is blocked or if the SPID is blocking other SPIDs, and if the waittime column shows a high number, you can decide to delete the SPIDIf the open_trans column has a value of 1 for a SPID line, there is a pending transaction for the SPID line. Additionally, if the waittime column shows a high number for this SPID line, you should delete the SPID

Finding User and computer from the AOT

Users should NOT end task (CTRL–ALT-DEL) their Dynamics AX session. Instead your users should:

Contact their AX administrator and have them end their session from the (Administration>Online) users form

<Or>

Users should CTRL-BREAK the process.

*This allows the AOS to gracefully close network connections and recover resources

RPC errors

RPC 1726 – communication issue between client and server. Typically it doesn’t mean you have a network error, but rather that the AOS was overloadedRPC 5 – Access deniedRPC 1702 -The binding handle is invalid

AOS Crash Maintenance

Options for collecting dump filesOption in Dynamics AX Server ConfigurationHow to get information to Microsoft

AOS Crashing options

Automatically send reports to MicrosoftHow to use ADPlus to troubleshoot "hangs" and "crashes” http://support.microsoft.com/kb/286350/en-usDebugging Tools for Windows – Overview (Download) http://www.microsoft.com/whdc/devtools/debugging/default.mspx Debug Diagnostic Tool http://www.microsoft.com/downloads/details.aspx?FamilyID=28bd5941-c458-46f1-b24d-f60151d875a3&displaylang=en

Automatically Send Report to Microsoft

Mark the “For all AOS instances in this computer, automatically send reports about fatal errors to Microsoft” option on the Application Object Server tab in the Server Configuration Utility.

Informing Microsoft of Crash

When AOS crashes you receive an event id of 1001 in application event logHere is an example of the error:

Microsoft Analyzes the Crash Dump File(s)

Contact Microsoft Support with contents of the application event log event id 1001Support Technician passes bucket id on to Escalation Engineer so they can obtain fileTechnical Support may contact you for more information. Ie. Application files (*.AOD)

Tools to troubleshoot AOS

Performance MonitorApplication event logLoggingSQL Server toolsDebug Diagnostic Tool

Performance Monitor

PAL toolMemory

Typically greatest cause of issuesMonitor VIRTUAL BYTES in 32 bit environmentMonitor WORKING SET in 64 bit environment

CPUPrior to Dynamics AX mostly an issue for batch processingIn 2009 with 64 bit CPU may see more CPU usage than in the past

Application Event Log

Monitor itHave a schedule for monitoring

Sizing of itMake sure events don’t get overwritten

Schedule to clear it out

Logging Tools

SQL Server Tracing in Dynamics AXEasy way to get sql and X++

Client/Server TracingMore advanced requires some setup and use of another tool to review information

Trace ParserUsed to review Client/Server Tracing

SQL Server tools

SQL ProfilerSee log of what sql is doing

SQL Management StudioRunning queriesChanging settingsMaintenance

SQL DMV’s

Debugging tools

Debug Diagnostic ToolHave it installed on AOS machinesHave Hang collection setup

Debugging tools for WindowsIf you want to review memory dump files

Q & A

© 2008 Microsoft Corporation. All rights reserved. Microsoft, Windows, Windows Vista and other product names are or may be registered trademarks and/or trademarks in the U.S. and/or other countries.

The information herein is for informational purposes only and represents the current view of Microsoft Corporation as of the date of this presentation. Because Microsoft must respond to changing market conditions, it should not be interpreted to be a commitment on the part of Microsoft, and Microsoft cannot guarantee the accuracy of any information provided after

the date of this presentation. MICROSOFT MAKES NO WARRANTIES, EXPRESS, IMPLIED OR STATUTORY, AS TO THE INFORMATION IN THIS PRESENTATION.