KACE Agent Architecture and Troubleshooting Overview

Dell World 2014

KACE Agent Architecture and Troubleshooting Overview

Allen Tsai: Principal Engineer

Rob Napier: Principal Engineer November, 06, 2014

Dell WorldUser Forum

Dell World 2014

• Changes in 6.0/6.3

• Key field issues addressed

• What a healthy agent looks like

• Data files

• Agent plugins

• Log file

• Windows installer

• AMP watchdog

• Agent security

• Debugging tips

Overview

Dell World 2014

• Application Blacklisting for Windows and Mac

• Updated User Alert (Windows and Mac parity)

• Expanded inventory collection

• Improved reliability

• Improved security

• AMP watchdog

Changes in 6.0 & 6.3

Dell World 2014

• KA-334 : Cannot replicate patches to UNC path

• KA-1231: Replicated large files keep on growing in size

• KA-1328: AMPAgent incorrectly terminates csrss.exe as its child process upon exit

• KA-231: AMPAgent can utilize too much CPU on Mac and Windows

Key Field Issues Addressed in 6.0/6.3

Dell World 2014

• AMPAgent service/daemon running

• Valid amp.conf with the proper host specified

• Valid amp_auto.conf (6.3) with AMP port 52230

• Network characteristics of a healthy agent

• Heartbeat every 20 seconds

• Regular inventory (interval specified by K1)

• Cycle thru read/write operations (visible in debug)

What a healthy agent looks like

Dell World 2014

• amp.conf

• amp_auto.conf (6.3)

• inventory.xml

• kinventory.db

• InventoryData.Software

• ksw_process.db

• ksw_timestamps.db

• Metering_data.txt

• kbots_cache/kbots.xml

Data Files

Dell World 2014

• host=<hostname> : all you need in amp.conf is a host field, and the agent can fill in the rest once it’s connected to the server.

• debug=true|all : use debug field to increase logging, but watch out for rollover.

• The log will roll over at 4 MB and 5 old logs are kept.

• Starting with 6.3, all the below calculated properties are stored in amp_auto.conf• wto, rto, cto, crto : write, read, connect and connect-retry timeouts

• servercompress : control whether the agent automatically compress uploads

• maxDownloadSpeed : set the max speed in KB/s that the agent can download payloads at

• processtimeout=xxx : override default process timeout in milliseconds

• ampurl, ampport, weburl, webport, companyname, splashtext, etc… : some variables are controlled by agent and should not be changed by the user

amp.conf & amp_auto.conf

Dell World 2014

• kinventory.db

• SQLite database used for inventory capture and for generating inventory.xml file.

• Useful for debugging when inventory.xml is missing or incomplete.

• Can be deleted to have kinventory repopulate in case of suspected corruption.

• inventory.xml

• The XML that describes the machine generated from the information in kinventory.db

• Useful to check if agent is collecting the information correctly when troubleshooting incorrect inventory data

kinventory.db & inventory.xml

Dell World 2014

• Software inventory collected to match against Dell Software Catalog

• Contains the list of all binaries and their attributes from the entire file system

• Contains some additional information such as Windows add/remove registry keys

• Used by K1 to determine all the software titles installed on the system

• Not to be confused with regular inventory, which is more hardware oriented

InventoryData.software

Dell World 2014

• SQLite database introduced in 5.5 to capture the software metering data.

• This database stores all the real time metering data as processes launch/terminate, and will be flushed when server asks for a report.

• Maximum rows is configurable. The default maximum is 5000 rows and can be adjusted depending on the software meter flush interval.

• On the Mac, there are multiple copies of ksw_process database, one per logged in user. The databases will have _username appended to the base filename.

• E.g.: ksw_process_atsai.db

ksw_process.db

Dell World 2014

• SQLite database introduced in 5.5 to store information such as last alive date

• Used to recover in case software metering process itself is terminated and unable to determine the termination time for process it is monitoring. Provides a best guess answer as to when processed we were monitoring terminated once software meter process starts back up.

• Like ksw_process.db, there are multiple DB on the Mac, one per user with _username appended to the base filename.

• E.g.: ksw_timestamp_atsai.db

ksw_timestamp.db

Dell World 2014

• Software meter results flushed from ksw_process.db

• Contains a list of all processes that ran on the system, their attributes, start time and end time as well as user that launched the process.

• This is generated when K1 asks the agent to flush its result and upload to server.

metering_data.txt

Dell World 2014

• Lists all online and offline scripts.

• Provides ID and VERSION of active scripts, located in the same folder.

• Used by the boot, login and offline script engines to loop through and look for applicable scripts.

• Updated by kbot number 3.

• C:\Program Files (x86)\Dell\KACE\runkbot.exe 3 0

kbots_cache/kbots.xml

Dell World 2014

• The primary functions of the agent arehandled by the AMPAgent service/daemon and 4 supporting plugins

• pluginWeb: handles script downloads, replication, log uploads.

• pluginRunProcess: handles scripts and runs processes thru runkbot.

• pluginPatching: handles detecting/deploying of patches.

• pluginDesktopAlerts: handles displaying broadcast user alerts or pre-install script alerts.

Agent Plugins

Dell World 2014

• With 6.0, we consolidated all agent logs into a single log file KAgent.log.

• This allows for a better trace of exact events that happened on the agent without having to cross reference all the logs and match up by time.

• The log contain the date, module and the function name that generated the log.

[2014-10-14.19:58:32][KInventory:CInventoryData::Initi] KInventory InventoryData opened DB successfully

[2014-10-14.19:58:32][KInventory:CInventoryData::Initi] KInventory InventoryData populated the DB

[2014-10-14.19:59:15][KInventory:runInventory ] KInventory Inventory Capture completed and stored in

C:\ProgramData\Dell\KACE\inventory.xml

[2014-10-14.19:59:15][KCopy:UploadUsingCurl ] UploadFile: Server gzip compression is active

[2014-10-14.19:59:15][KCopy:UploadUsingCurl ] UploadFile: uploading file C:\ProgramData\Dell\KACE\inventory.xml.gz to

https://engk1agent3/service/inventory.php?KUID=F2C603AD-08C8-48D3-A556-25F2702F6D89&VERSION=6.0.32

Log File starting 6.0

Dell World 2014

• By default, Windows agent provisioning, removal and updates produce two debug logs files:

• ampmsi.log: msiexec /L*v log file, found in the %TEMP% folder

• ampinstaller.log: custom action log file, found in the %TEMP% folder as well as the KACE data folder

• The log files indicate when agent installed and/or uninstalled, and flags and properties passed in.

• When run non-silently, the agent will honor the current locale language if is one of the 10 supported languages. Otherwise, default is English.

• Use the new GPO Provisioning tool to help create the GPO. Will not need to use the setlang VB script to override default locale anymore if use the GPO Provisioning tool.

• https://www.kace.com/support/resources/kb/solutiondetail?sol=133776

Windows Installer

Dell World 2014

• /i <msi file> : install msi file (example: msiexec /i amp.msi)

• /x <msi file> : The preferred way to uninstall agent is to run “AMPTools uninstall”

• /qn : silent install (example: msiexec /qn /i amp.msi)

• /L*v <log_file> : create log file (example: msiexec /L*v amp.log /i amp.msi)

• HOST=<host_name> : set amp.conf host value (example: msiexec /i amp.msi HOST=kbox7.acme.com)

• Alternatively: you can append to msi filename (example: msiexec /i amp_kbox7.acme.com)

• DEBUG=true : set amp.conf debug value (example: msiexec /i amp.msi HOST=kbox7 DEBUG=true)

• NOHOOKS=1 : don’t install boot & logon hook dlls (msiexec /i amp.msi HOST=kbox7 NOHOOKS=1)

• CLONEPREP=1 : do not start AMP service until next reboot (msiexec /i amp.msi CLONEPREP=1)

Windows Installer Common Properties

Dell World 2014

• Introduced in 6.3, AMPWatchDog monitors the health of agent and perform simple recovery to address majority of the common issues

• Conditions which AMPWatchDog monitors currently:

• AMPAgent executable exists in expected location

• AMP configuration file exist with server host

• AMP Service/Daemon not running (Restart agent)

• Inventory.xml exists and is less than 3.5 times the configured frequency (Restart agent)

• ampport=port exists in amp_auto.conf (Restart agent)

• Logs netstat output relevant to ampagent connection

AMPWatchDog

Dell World 2014

• Agent by default will accept http connection if https can not be established with K1

• Uses OpenSSL 1.0.1h with Heartbleed addressed

• Agent is not affected by the POODLE vulnerability

• Only publicly signed certificates honored

• The signing root authority must be included in the curl certificate bundle (cacert.pemin agent data directory, exported from Mozilla)

• Different SSL options to set in amp.conf

• sslrequired: Agent will not fall back to http if https can not be established

• verifyssl: Agent will verify the server certificate, implied sslrequired

• TLS will be on by default in the future

Agent Security

Dell World 2014

• What are the basic stuff to look for when suspecting agent issues?

• Turn on Debug=true

• Is agent running?

• Is the KAgent.log being updated? (under users directory)

• Is the agent connected? DNS issues? Use FQDN when possible. Can you resolve the host and telnet to port 52230?

• Agent is 32 bits, remember that for any registry/file system OS redirections

• Test agent functionalities by running runkbot 2 0

• Windows: c:\Program Files (x86)\Dell\KACE\runkbot 2 0

• Mac: /Library/Application Support/Dell/KACE/bin/runkbot 2 0

• Linux: /opt/dell/kace/bin/runkbot 2 0

• Are there crash dumps?

• Windows: c:\ProgramData\Dell\KACE\*.dmp

• Mac: Console

Trouble Shooting

Dell World 2014

• Eliminate possible system conflicts (disable firewall, turn off AV program, etc.)

• Look at the log file in the KACE data folder

• Win XP: “C:\Documents and Settings\All Users\Dell\KACE\user

• Vista and Win 7: C:\ProgramData\Dell\KACE\user

• Mac: /Library/Application\ Support/Dell/KACE/data/user

• Linux: /var/dell/kace/user

• Verify valid amp.conf file, and regenerate it if needed using AMPTools

• AMPTools resetconf host=kbox7.acme.com

• Verify existence of valid kbots_cache/kbots.xml files and supporting script xml files

Debugging Tips

Dell World 2014

• Verify K1000 host name resolves using browser or command line

• ping kbox7.acme.com

• telnet kbox7.acme.com 52230

• Enable debugging by running “AMPTools debug=true” which will set debug value in amp.conf and restart the agent.

• Alternatively, you can temporarily enable debugging on Windows without restarting the agent using “sc control ampagent 199” (This only enables debug for AMPAgent, not all binaries)

Debugging Tips (continued)

Dell World 2014

• HexDump

• Setting HEX_DMP environment variable before starting AMPAgent will cause AMPAgent to log the exact informtion it tries to send and receive over the wire.

• HeartBeato [Smurf_write_SYNC ] -------------------------------------------------------------

------------------- [Smurf_write_SYNC ] 00000000 00 00 00 01 05..... [Smurf_write_SYNC ] 00000005 ---------------------------------------------------------------------

• Agent connection stringo [Smurf_write_SYNC ] -------------------------------------------------------------

-------------------

o [Smurf_write_SYNC ] 00000000 00 00 00 01 02 00 06 00 00 00 21 36 34 34 35 46..........!6445F

o [Smurf_write_SYNC ] 00000010 45 42 34 34 34 33 37 34 36 33 31 41 30 33 39 35EB444374631A0395

o [Smurf_write_SYNC ] 00000020 30 42 31 31 32 34 39 37 34 31 41 00 00 00 00 0a0B11249741A.....

o [Smurf_write_SYNC ] 00000030 30 2c 32 2c 34 2c 36 2c 35 00 00 00 00 6b 76 650,2,4,6,5....kve

Debugging Tips (continued)

Dell World 2014

• Basic information to collect when observed crashes

• Collect crash dumps or crash call stack

• Windows 2008/Vista and newer: HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\Windows\Windows Error Reporting\LocalDumps

• Mac: Agent crashes will show under System Diagnostic Report in Console. Collect the crash call stack.

• Collect agent version

• Collect agent log in debug mode

• All the information are important in order to debug and simulate the failure successfully. Need the agent version in order to match the crash dump up with the debug symbols.

Crashes Diagnostics

Dell World User Forum

Q & A

Dell World User Forum

Thank you.