SYN402: Advanced Tools and Techniques for Troubleshooting NetScaler Appliances Andrew Redman | Lead Escalation Engineer May 8, 2014
Jan 15, 2015
SYN402: Advanced Tools and Techniques for Troubleshooting NetScaler AppliancesAndrew Redman | Lead Escalation Engineer
May 8, 2014
© 2014 Citrix. Confidential.2
Tweet about this session with hashtag #SYN402 and #citrixsynergy
© 2014 Citrix. Confidential.3
NetScaler System Overview
Troubleshooting Tools & Techniques
Case Studies
Resources
Conclusion
Q&A
Agenda
NetScaler System Overview
© 2014 Citrix. Confidential.5
Key NetScaler Processes
ns_master/NSPPEnsvpndnsaaadnsconf
nsauthdnslog.sh
nssyncnsreadfile
nslcdnsfsyncdnsnetsvc
nsconmsgnscollect
Runs Citrix NetScaler OSSSL VPN File TransferRBA and SSL VPN external authorizationWrites the ns.conf fileCLI authenticationControls logging for the newnslogHA synchronizationUsed to read SSL certificate filesRuns the front panel LCDSynchronizes bookmarks and SSL certificatesUsed by the GUI for configuration changesControls writing of the newnslogStatistics gathering for historical purposes
Process Description
© 2014 Citrix. Confidential.6
NetScaler File System
/var (hard drive) Logs - /var/log & /var/nslog
Install - /var/nsinstall
Trace - /var/nstrace
Core Dumps - /var/crash & /var/core
/flash (flash drive) Config - /flash/nsconfig
SSL Certificates - /flash/nsconfig/ssl
© 2014 Citrix. Confidential.7
NetScaler File System (cont.)
/flash (cont.) User Monitors - /flash/nsconfig/monitors
Custom Options - /flash/nsconfig
/ (ram drive) OS - (operating system)
Troubleshooting Tools & Techniques
NetScaler
> show techsupport
Critical System Data
In-Depth Performance Monitoring
Stats
Detailed Log Files
USER Command Logging
/var/tmp/support/collector_P_10.10.10.10_21Apr2014_21_42_tar.gz
NetScaler Tech Support Bundle
© 2014 Citrix. Confidential.10
The NetScaler Tech Support Bundle
© 2014 Citrix. Confidential.11
Citrix Predictive Support
© 2014 Citrix. Confidential.12
‘Single Mission … Data Collection’ FAQ: http://support.citrix.com/article/CTX131233
Data Collection Analyze DataTailored
Recommendations
© 2014 Citrix. Confidential.13
Use this box as a guideline for proper image placement. Place your image on top and align to box.
https://taas.citrix.com
© 2014 Citrix. Confidential.14
Technical Issues Flagged
© 2014 Citrix. Confidential.15
Investigate Issues Further
The ‘BELL’
© 2014 Citrix. Confidential.16
Crash File(s) Location
nscollect-542.gz
© 2014 Citrix. Confidential.17
Intuitive Navigation
Select a different newnslog file to view
© 2014 Citrix. Confidential.18
Detailed Graphs
Mouse over graph to see more detail
Informative reference legend
Download the data as an excel sheet
© 2014 Citrix. Confidential.19
IPMI – Intelligent Platform Management Interface
Default LOM IP Address: http://192.168.1.3
Change NetScaler IP Address
Obtain Health Monitoring Detail
Harvest Serial Number
Determine MAC Address
© 2014 Citrix. Confidential.20
Common CLI Show Commands
Common show commands for system information:
show node, show info, show license
Common show commands for vserver and service:
show lb vserver, show cs vserver, show service, show persistencesession
show connectiontable
Other common show commands:
show route, show ip
© 2014 Citrix. Confidential.21
Common CLI Stat Commands
Common stat commands for system information:
stat ns, stat cpu, stat interface
Common stat commands for vserver and service:
stat lb vserver, stat cs vserver, stat service
Other common stat commands:
stat dns, stat ssl, stat http
© 2014 Citrix. Confidential.22
Leveraging ‘nsconmsg’
Nsconmsg common use cases:
View events
View console messages
View statistics
Debug system counters
Debug load balancing issues
Debug CPU/Memory utilization
Make absolutely sure that you
use a capital -K and NOT a
lower-case -k
© 2014 Citrix. Confidential.23
Example ‘nsconmsg’ Usage
# cd /var/nslog
# nsconmsg -K newnslog -j fqdn-ssl-vip -s ConLb=1 -d oldconmsg
current log file
newnslog
name of vserver
fqdn-ssl-vip
LB stats
ConLb=1
© 2014 Citrix. Confidential.24
Displaying debug performance informationNetScaler V20 Performance DataNetScaler NS10.1: Build 123.11.nc, Date: Feb 24 2014, 17:30:43
current time is Sun Mar 23 18:33:43 2014-------------------------------------------------------NATSession : Free(6553)A(6553)InUse(0)NATSession: Cur(Tcp[0] Udp[0] Icmp[0] Other[0])NATSession: Op/s(Tcp[0] Udp[0] Icmp[0] Other[0])Session: A:0 F:0 IUse:0 SEs: SIP:0 C:0 SSL:0 Svr:0 UserId:0 SIPDIP:0 DIP:0 SO:0SSF: Conn (Srvr 0 Clnt 0) U:0CM: Conn (Srvr 0 Clnt 0) Sessions PCB 0 NATPCB 0Z(SIP[0], C[0], SSL[0] Server[0] SIPDIP[0] DIP[0] SO[0])Mon: Probes: 434562009, Failed: 15VIP(10.54.169.75:443:UP:LEASTCONNS): Hits(7317, 0/sec) Mbps(0.00) Pers(OFF) Err(0) SO(0) LConn_BestIdx: 0S(10.54.148.201:80:UP) Hits(7317, 0/sec, P[0, 0/sec]) ATr(0:0) Mbps(0.00) BWlmt(0 kbits) RspTime(0.00 ms) Load(0) LConn_Idx: (C:0; V:0,I:1)-------------------------------------------------------CPU:0.2% MEM:182472560 UP:10.00:00:38 since:Thu Mar 13 18:33:05 2014
current time is Sun Mar 23 18:33:43 2014
Mon: Probes: 434562009, Failed: 15VIP(10.54.169.75:443:UP:LEASTCONNS): Hits(7317, 0/sec) Mbps(0.00) Pers(OFF)S(10.54.148.201:80:UP) Hits(7317, 0/sec, P[0, 0/sec]) ATr(0:0) Mbps(0.00) BWlmt(0 kbits) RspTime(0.00 ms)
CPU:0.2% MEM:182472560 UP:10.00:00:38 since:Thu Mar 13 18:33:05 2014
© 2014 Citrix. Confidential.25
# nsconmsg -K newnslog -j <name of VIP> -s ConLb=1(2 or 3) -d oldconmsg | more
# nsconmsg -K newnslog -s ConMon=1 -d oldconmsg
# nsconmsg -K newnslog -s ConMEM=1 -d oldconmsg
# nsconmsg -K newnslog -s ConSSL=1 -d oldconmsg
ConDebug - DebuggingConLb - Load BalancingConMon - Monitoring ProbesConMEM - Memory ManagementConCSW - Content SwitchingConSSL - SSL OffloadConCMP - CompressionConIC - Integrated Caching
© 2014 Citrix. Confidential.26
Log File Analysis
# cd /var/log
ns.log:Mar 20 16:45:06 <local0.info> 10.54.169.73 03/20/2014:20:45:06 GMT atlvpx 0-PPE-0 : UI CMD_EXECUTED 2947 0 : User nsroot - Remote_ip 10.13.73.65 - Command "login nsroot "********"" - Status "Success"
ns.log:Mar 20 16:45:06 <local0.info> 10.54.169.73 03/20/2014:20:45:06 GMT atlvpx 0-PPE-0 : UI CMD_EXECUTED 2948 0 : User nsroot - Remote_ip 10.13.73.65 - Command "show ns license" - Status "Success"
# zgrep -i cmd_executed ns.log* | more (the -i means ignore CASE)
interface down vServer down panic signaled
© 2014 Citrix. Confidential.27
NetScaler + Wireshark = ‘thumbs up’http://www.wireshark.org
© 2014 Citrix. Confidential.28
RED HOT Wireshark Tip
Custom Columns
Custom Menu Options
Pre-build Custom Filters
Much Faster Analysis
Get the red hot details on how to empower your default Wireshark configuration in the Reference Section at the end of this presentation.
Troubleshooting Techniques & Case Studies
© 2014 Citrix. Confidential.30
Top Tips
Use Citrix Predictive Support
Pay attention to the issues!
Note the highlighted counter(s)
Use nsconmsg to see even more detail
Correlate time-frames in other log files
Achieve root cause analysis faster!
RCA
Achieved!
Case #1 - High Availability Synchronization
© 2014 Citrix. Confidential.32
Primary NetScaler Secondary NetScaler
Same type of appliance
Same firmware version
Same nsroot password
Same RPC Node password
Open requisite TCP ports
HA Pre-requisites
© 2014 Citrix. Confidential.33
Predictive Support Flags The Issues
© 2014 Citrix. Confidential.34
The HA Pair Struggled To Synchronize
# nsconmsg -K newnslog -d statswt0 | grep nic_tot_bdg_mac_moved (nic_err_bdg_muted)
57520 0 71837018 nic_tot_bdg_mac_moved interface(0/1)
57521 0 71837018 nic_tot_bdg_mac_moved interface(0/2)
9861 0 65 nic_err_bdg_muted interface(0/1)
9862 0 65 nic_err_bdg_muted interface(0/2)
71,837,018 MAC Moves
65 Interface Mutes
© 2014 Citrix. Confidential.35
The ‘newnslog’ Time-Frame
# nsconmsg -K newnslog -d setime
Displaying start and end time information
NetScaler V20 Performance Data
NetScaler NS9.3: Build 54.4.nc, Date: Dec 20 2011, 22:44:41
start time Fri Feb 28 21:49:58 2014
end time Fri Feb 28 21:53:28 2014
total duration 00.00:03:30
data size 1,718,949 bytes
total duration 00.00:03:30
Case #2 - XA/XD Slow Performance
© 2014 Citrix. Confidential.37
XenAppTablets
Smartphones XenDesktop
Predictive Support
Critical Insight Gleaned
Preventative Approach
Don’t Underestimate
XA/XD Slow Performance
© 2014 Citrix. Confidential.38
Predictive Support Flags The Issues
http://support.citrix.com/article/CTX136926
© 2014 Citrix. Confidential.39
Performance Was Extremely Latent
# nsconmsg -K newnslog -d statswt0 | grep nic_tot_bdg_mac_moved
4263 0 23 nic_tot_bdg_mac_moved interface(0/1)
4264 0 51 nic_tot_bdg_mac_moved interface(1/1)
4265 0 28 nic_tot_bdg_mac_moved interface(1/2)
23, 51 & 28 MAC Moves
© 2014 Citrix. Confidential.40
Networking Issues Again?
# nsconmsg -K newnslog -d statswt0 | grep nic_err
4274 0 1995 nic_err_rl_pkt_drops interface(1/1)
4275 0 40736 nic_err_rl_pkt_drops interface(1/2)
4276 0 1995 nic_err_rl_rate_pkt_drops interface(1/1)
4277 0 40736 nic_err_rl_rate_pkt_drops interface(1/2)
4678 0 42731 allnic_err_rl_rate_pkt_drops
System Limits Exceeded
Rate-limited Packets!
© 2014 Citrix. Confidential.41
The Moral of the Story Leverage Citrix Predictive Support
LeveragePred. Support
Pay AttentionGain Quick
InsightDig Into
‘nsconmsg’On Target for
Success!
Resources
© 2014 Citrix. Confidential.43
Helpful Resources
Comprehensive NetScaler Counters
Wireshark Developer Editions
Customizing Wireshark Tutorial
Citrix Predictive Support Forum
NSTRACE Options
How To Manage VLAN’s, Interfaces and Subnets
Conclusion
© 2014 Citrix. Confidential.45
What We’ve Actually Covered
An Overview of the NetScaler System to give you a high-level understanding of the core system.
I shared with you some excellent Troubleshooting Tools that are available at your disposal.
I also discussed a few key Troubleshooting Techniques that you can use to diagnose issues.
I then highlighted two different Case Studies leveraging the tools & techniques that I shared with you in the presentation.
In addition I provided you with a few Resources for your future reference and edification.
Q & A
© 2014 Citrix. Confidential.47
Before you leave…
Conference surveys are available online at www.citrixsynergy.com starting Thursday, May 8 at 9:00 a.m.
Provide your valued feedback by 6:00 p.m. today to be entered to win one of many prizes!
Download presentations starting Monday, May 19 from the My Event Planning tool
© 2014 Citrix. Confidential.48
WORK BETTER. LIVE BETTER.