© 2014 VMware Inc. All rights reserved. VMware vRealize Automation 6.1/6.2 Logging Overview and Configuration for Log Insight Content Pack Kimberly Delgado, @KCDAutomate Steve Flanders, @smflanders April 2015
Welcome message from author
This document is posted to help you gain knowledge. Please leave a comment to let me know what you think about it! Share it to your friends and learn new things together.
Transcript
© 2014 VMware Inc. All rights reserved.
VMware vRealize Automation 6.1/6.2 Logging Overview and Configuration for Log Insight Content Pack
Kimberly Delgado, @KCDAutomate Steve Flanders, @smflanders
April 2015
Updates • Content Pack
– April 2015 – v1.1 includes updates to vRO queries to account for new log format introduced in 6.0.1, and update to General Overview, vRA events over time
• Document – April 2015 – updates to Agent doco links and instructions, and updates to vRO Log
config
2
Distributed Architecture
3
SSO
vRA VAs
vRO
Infrastructure Web / Manager LBs
Infrastructure Web / Manager Servers
Infrastructure Agents/DEMs
Clustered SQL Server DB
Infrastructure Fabric
NOTE: see the vRA Reference Architecture guide for detailed layouts.
vRA App Svcs
Host Roles • SSO: Authentication from vCenter 5.5 SSO, SSO Identity Appliance or the
SSO standalone Windows host
• vRA Virtual Appliance(s): host the CAFÉ & Code Stream services, embedded Postgres DB instance and embedded vRO Instance; CAFÉ services can be configured in distributed manner on multiple instances of the VA
• vRO (External): for non-POC environments, an external vRO configuration can be standalone or load-balanced with external DB
• App Services: VA for application services components
• Infrastructure Web Server: hosts web server UI, WAPI interface and Model Manager
• Manager Service: responsible for moving Infrastructure components through their defined lifecycle
• DEMs: Orchestrator & Workers – interacts with Fabric sources
• Agents – interacts with Fabric sources
4
Remote Logging • Operating Systems
– Windows does not natively support syslog – Virtual Appliances (VAs) and Linux do support syslog
• Log Insight Agent – Available for both Windows and Linux – Easy to deploy and configure; very lightweight – Ability to handle multiline messages and tag events – Properly handles log spikes and log rotation – Offers capabilities beyond those provided by syslog
• Use of the Log Insight agent is recommended for all vRealize Automation components (Windows and Linux) except vRO
• The Log Insight agent configurations include custom Tags which are leveraged in the vRA content pack. If not properly configured, some queries may not work as expected.
5
Remote Logging, continued… • The Log Insight agents can be downloaded directly from Log Insight,
under Administration / Management / Agents and follow the download link at the bottom of the page.
• Log Insight Windows agent installation instructions:
http://pubs.vmware.com/log-insight-25/topic/com.vmware.log-insight.administration.doc/GUID-455106F4-4C3D-47C1-8EF6-84992BCCEB05.html
• Log Insight Linux agent installation instructions (for the vRealize virtual appliances, choose RPM):
http://pubs.vmware.com/log-insight-25/topic/com.vmware.log-insight.administration.doc/GUID-DB4A27CF-BDA7-443F-94FB-AB9097AD8008.html
6
vRA 6.1+ Content Pack If you haven’t done so already, you can download the vRealize Automation 6.1+ Content Pack in one of two ways:
• Logged into Log Insight as an Administrator, navigate to Content Packs / Marketplace. The VMware - vRA 6.1+ content pack will be near the bottom of the page.
• Go to the VMware Solutions Exchange (https://solutionexchange.vmware.com/) and navigate to Cloud Management Marketplace, VMware vRealize Log Insight. You will need to scroll to locate or search for “vRA”.
7
Log Locations
vCenter SSO • Identity VA (SSO VA & vCenter SSO)
– /var/log/vmware/sso/* • Catalina.out (primary) • ssoAdminServer.log – user log in info here • vmware-identity-sts-perf.log • vmware-identity-sts.log • vmware-sts-idmd-perf.log • vmware-sts-idmd.err • vmware-sts-idmd.log
– /var/log/messages – Active Directory connection info
• Windows VIM on vCenter SSO – C:\ProgramData\VMware\CIS\logs\vmware-sso\ – C:\ProgramData\VMware\CIS\runtime\VMwareSTS\logs
9
vRealize Automation – Virtual Appliances • vRealize Automation (vRA) & Code Stream (vRCS)
– /var/log/vmware/vcac/catalina.out
– /var/log/apache2/access_log
– /var/log/apache2/ssl_request_log
– /var/log/apache2/error_log
• vRealize Orchestrator (embedded & external same location): – /var/log/vco/app-server/catalina.out
– /var/log/vco/app-server/server.log
– /var/log/vco/app-server/scripting.log
– Individual plugins need to be configured for logging and may have different log locations
• Application Services – /home/darwin/tcserver/darwin/logs/catalina.out
• Artifactory (for Code Stream) – /storage/artifactory/home/logs/artifactory.log
– /storage/artifactory/home/logs/access.log
– /storage/artifactory/home/logs/request.log
– /storage/artifactory/home/logs/import.export.log
10
vRealize Automation – Infrastructure • Exact logs & locations will depend on deployment type and
configuration; these are basic places to start!
• Infrastructure Server (Web, Manager) – C:\Program Files (x86)\VMware\vCAC\Server\Logs\All – C:\Program Files (x86)\VMware\vCAC\Server\Config Tool\Log\vCACConfiguration-
<date> – C:\Program Files (x86)\VMware\vCAC\Server\Model Manager Data\Logs\ – C:\Program Files (x86)\VMware\vCAC\Server\Model Manager Web\Logs\Repository – C:\Program Files (x86)\VMware\vCAC\Website\Logs\Web_Admin_All – C:\Program Files (x86)\VMware\vCAC\Web API\Logs\
11
vRealize Automation – Infrastructure, continued… • Some log directories and filenames are set during installation and will
depend on entered information. Information like <THIS> needs to be replaced with entered information.
• Agents – C:\Program Files (x86)\VMware\vCAC\Agents\<PLUGIN>\logs\<FILE>
• <PLUGIN> Examples: vSphereAgent, nsx, VC55Agent, VDIAgent • <FILE> Examples: vSphereAgent, EpiPowerShellAgent, VdiPowerShellAgent IMPORTANT: The Agent name specified during installation dictates the value of <PLUGIN>
• DEMs – C:\Program Files (x86)\VMware\vCAC\Distributed Execution Manager\<DEM_NAME> – C:\Program Files (x86)\VMware\vCAC\Distributed Execution Manager\<DEO_NAME> – IMPORTANT: The DEM/DEO name specified during installation dictates the value
12
vRealize Business • vRB Data Collector
– /var/log/itbm-data-collector/catalina.out – /var/log/itbm-data-collector/itfm-vc-dc.log – /var/log/itbm-data-collector/localhost_access_log.* – /var/log/itbm-data-collector/vf.tc-events.txt
• vRB Server – /var/log/itbm-server/audit.log – /var/log/itbm-server/catalina.out – /var/log/itbm-server/itfm-external-api.log – /var/log/itbm-server/itfm-reflib-update.log – /var/log/itbm-server/itfm.log – /var/log/itbm-server/localhost_access_log.* – /var/log/itbm-server/vcac.log – /var/log/itbm-server/vf.tc-events.txt
13
Syslog Configuration
Log Insight Server-Side Agent Configuration • Log Insight agent configuration can be set client-side or server-side.
Server-side consists of three steps outlined below. The slide following have client-side configurations.
1. Enable vRO logging – see the vRO slide for configuration information
2. Static configuration (copy and paste): ;;; vCenter SSO VCSA [filelog|vmw-sso] directory=/var/log/vmware/sso exclude=vmware-* event_marker=^(\[\d{4}-\d{2}-\d{2}|\d{2}-\w+-\d{4}) tags={"vmw_product":"sso"}
[filelog|vmw-sso-sts-idmd-perf] directory=/var/log/vmware/sso include=vmware-sts-idmd-perf* event_marker=^\d{4}-\d{2}-\d{2}\s\S+\s\w+\s+\w+ tags={"vmw_product":"sso"}
[filelog|vmw-sso-sts-perf] directory=/var/log/vmware/sso include=vmware-identity-sts-perf* event_marker=^\[\d{4}-\d{2}-\d{2}\s\S+\s\S+\s\S+\]\s+\w+ tags={"vmw_product":"sso"}
[filelog|vmw-sso-sts-other] directory=/var/log/vmware/sso include=vmware-sts-idmd.*;vmware-identity-sts.* event_marker=^\[\d{4}-\d{2}-\d{2}\s\S+\s\S+\s\S+ tags={"vmw_product":"sso"}
15
Log Insight Server-Side Agent Configuration ;;; vCenter SSO Windows [filelog|vcenter-sso] directory=C:\ProgramData\VMware\CIS\logs\vmware-sso event_marker=^\[\d{1,2}/\d{1,2}/\d{4} \d{1,2}:\d{1,2}:\d{1,2} tags={"vmw_product":"sso"} [filelog|vcenter-sso-sts] directory=C:\ProgramData\VMware\CIS\runtime\VMwareSTS\logs event_marker=^\[\d{1,2}/\d{1,2}/\d{4} \d{1,2}:\d{1,2}:\d{1,2} tags={"vmw_product":"sso"}
;;; vRA
[filelog|vra] directory=/var/log/vmware/vcac event_marker=^\d tags={"vmw_product":"vra","vmw_product_component":"cafe"}
[filelog|apache] directory=/var/log/apache2 tags={"asf_product":"http"}
;;; vRCS
[filelog|vrcs] directory=/storage/artifactory/home/logs event_marker=^\d tags={"vmw_product":"vrcs","vmw_product_component":"jfrog"}
;;; vRA APPD [filelog|vra-appd] directory=/home/darwin/tcserver/darwin/logs event_marker=^\w+\s\d{2}\s\d{4}\s\S+\s\w+\s+[\S+] tags={"vmw_product":"vra","vmw_product_component":"appd"}
16
Log Insight Server-Side Agent Configuration ;;; Static vRA [filelog|vra-agent-vsphere] directory=C:\Program Files (x86)\VMware\vCAC\Agents\vSphereAgent\logs\ event_marker=^\[\d{1,2}/\d{1,2}/\d{4} \d{1,2}:\d{1,2}:\d{1,2} tags={"vmw_product":"vra","vmw_product_component":"agent"} [filelog|vra-server] directory=C:\Program Files (x86)\VMware\vCAC\Server\Logs\ include=*All.log;Repository.log event_marker=^\[\w\w\w:\d{4}-\d{2}-\d{2} tags={"vmw_product":"vra","vmw_product_component":"server"} [filelog|vra-mm] directory=C:\Program Files (x86)\VMware\vCAC\Server\Model Manager Web\Logs\ include=*All.log;Repository.log event_marker=^\[\w\w\w:\d{4}-\d{2}-\d{2} tags={"vmw_product":"vra","vmw_product_component":"mm"} [filelog|vra-web] directory=C:\Program Files (x86)\VMware\vCAC\Server\Website\Logs\ include=*All.log;Repository.log event_marker=^\[\w\w\w:\d{4}-\d{2}-\d{2} tags={"vmw_product":"vra","vmw_product_component":"web"} [filelog|vra-install] directory=C:\Program Files (x86)\VMware\vCAC\InstallLogs\ include=*All.log;Repository.log event_marker=^\[\w\w\w:\d{4}-\d{2}-\d{2} tags={"vmw_product":"vra","vmw_product_component":"install"}
17
Log Insight Server-Side Agent Configuration ;;; vRB
[filelog|vra-vrb-server] directory=/var/log/itbm-server event_marker=^\d tags={"vmw_product":"vrb","vmw_product_component":"server"}
[filelog|vra-vrb-data-collector] directory=/var/log/itbm-data-collector event_marker=^\d tags={"vmw_product":"vrb","vmw_product_component":"data-collector"}
3. Dynamic configuration (modify everything like <THIS>) because folder names are set based on install parameters provided:
;;; Dynamic vRA agent configuration ;;; MANUAL CONFIGURATION CHANGES REQUIRED ;;; DO NOT JUST COPY AND PASTE THIS SECTION ;;; For every agent installed a new agent configuration section is required ;;; The name of the agent given during installation dictates the log directory name [filelog|vra-agent-<AGENT_NAME>] directory=C:\Program Files (x86)\VMware\vCAC\Agents\<AGENT_NAME>\logs\ event_marker=^\[\d{1,2}/\d{1,2}/\d{4} \d{1,2}:\d{1,2}:\d{1,2} tags={"vmw_product":"vra","vmw_product_component":"agent"}
18
Log Insight Server-Side Agent Configuration ;;; A DEM name can be specified during installation ;;; The name of the DEM given during installation dictates the log directory name ;;; If no name is given the DEM name is: DEM [filelog|vra-dem] directory=C:\Program Files (x86)\VMware\vCAC\Distributed Execution Manager\<DEM_NAME>\Logs\ include=*All.log;Repository.log event_marker=^\[\w\w\w:\d{4}-\d{2}-\d{2} tags={"vmw_product":"vra","vmw_product_component":"dem"} ;;; A DEO name can be specified during installation ;;; The name of the DEO given during installation dictates the log directory name ;;; If no name is given the DEO name is: DEO [filelog|vra-deo] directory=C:\Program Files (x86)\VMware\vCAC\Distributed Execution Manager\<DEO_NAME>\Logs\ include=*All.log;Repository.log event_marker=^\[\w\w\w:\d{4}-\d{2}-\d{2} tags={"vmw_product":"vra","vmw_product_component":"deo"}
19
vCenter SSO on VCSA • Log Insight agent configuration (recommended – copy and paste): ;;; vCenter SSO VCSA [filelog|vmw-sso] directory=/var/log/vmware/sso exclude=vmware-* event_marker=^(\[\d{4}-\d{2}-\d{2}|\d{2}-\w+-\d{4}) tags={"vmw_product":"sso"}
[filelog|vmw-sso-sts-idmd-perf] directory=/var/log/vmware/sso include=vmware-sts-idmd-perf* event_marker=^\d{4}-\d{2}-\d{2}\s\S+\s\w+\s+\w+ tags={"vmw_product":"sso"}
[filelog|vmw-sso-sts-perf] directory=/var/log/vmware/sso include=vmware-identity-sts-perf* event_marker=^\[\d{4}-\d{2}-\d{2}\s\S+\s\S+\s\S+\]\s+\w+ tags={"vmw_product":"sso"}
[filelog|vmw-sso-sts-other] directory=/var/log/vmware/sso include=vmware-sts-idmd.*;vmware-identity-sts.* event_marker=^\[\d{4}-\d{2}-\d{2}\s\S+\s\S+\s\S+ tags={"vmw_product":"sso"}
• Syslog configuration (restart syslog after changes): – /etc/syslog-ng/syslog-ng.conf – Set "destination logserver" to syslog host or Log Insight
20
vCenter SSO on Windows • Log Insight agent configuration (copy and paste): ;;; vCenter SSO Windows [filelog|vcenter-sso] directory=C:\ProgramData\VMware\CIS\logs\vmware-sso event_marker=^\[\d{1,2}/\d{1,2}/\d{4} \d{1,2}:\d{1,2}:\d{1,2} tags={"vmw_product":"sso"} [filelog|vcenter-sso-sts] directory=C:\ProgramData\VMware\CIS\runtime\VMwareSTS\logs event_marker=^\[\d{1,2}/\d{1,2}/\d{4} \d{1,2}:\d{1,2}:\d{1,2} tags={"vmw_product":"sso"}
21
vRealize Orchestrator (vRO) • Syslog configuration: Edit /etc/vco/app-server/log4j.xml
– If Syslog was configured in VAMI, you MUST change the ”vco: prio:" section below highlighted in red to match how it is shown here, or you will get no vRO logs from the content pack queries!
– Edit section (remove comments and substitute <HOST> with Syslog or Log Insight host):
<appender name="SYSLOG" class="org.apache.log4j.net.SyslogAppender"> <param name="Threshold" value="INFO"/> <param name="Facility" value="LOCAL1"/> <param name="SyslogHost" value="<HOST>"/> <param name="FacilityPrinting" value="false"/> <layout class="org.apache.log4j.PatternLayout"> <param name="ConversionPattern" value="vco: prio:%-5p thread:%t token:%X{token} wf:%X{workflowName} wfid:%X{workflow} user: %X{username} cat:%c{1} msg:%m%n"/> </layout> </appender>
• At end of config xml (/etc/vco/app-server/log4j.xml) – Edit section (remove comments for SYSLOG appender):
<root> <priority value="INFO" /> <appender-ref ref="CONSOLE" /> <appender-ref ref="FILE" /> <appender-ref ref="SYSLOG" /> <!-- <appender-ref ref="EVENT_LOG" /> --> </root>
22
vRealize Automation (vRA) & Code Stream (vRCS) • Log Insight agent configuration (recommended – copy and paste): ;;; vRA
[filelog|vra] directory=/var/log/vmware/vcac event_marker=^\d tags={"vmw_product":"vra","vmw_product_component":"cafe"}
[filelog|apache] directory=/var/log/apache2 tags={"asf_product":"http"}
;;; vRCS
[filelog|vrcs] directory=/storage/artifactory/home/logs event_marker=^\d tags={"vmw_product":"vrcs","vmw_product_component":"jfrog"}
• Syslog configuration (restart syslog after changes):
– /etc/rsyslog.d/remote.conf – Add details for each log file (substitute <HOST> with Syslog or Log Insight
host at end):
23
vRealize Automation, continued… # # vRA + vRCS log files # Add to: /etc/rsyslog.d/remote.conf # Replace with Log Insight FQDN # Run: /etc/init.d/syslog restart # $ModLoad imfile # vRA $InputFileName /var/log/vmware/vcac/catalina.out $InputFileTag vcac: $InputFileStateFile stat-vcac-catalina1 $InputFileSeverity info $InputFileFacility local7 $InputRunFileMonitor $InputFileName /var/log/vco/app-server/catalina.out $InputFileTag vco: $InputFileStateFile stat-vco-catalina1 $InputFileSeverity info $InputFileFacility local7 $InputRunFileMonitor $InputFileName /var/log/apache2/access_log $InputFileTag apache: $InputFileStateFile stat-apache2-access1 $InputFileSeverity info $InputFileFacility local7 $InputRunFileMonitor $InputFileName /var/log/apache2/error_log $InputFileTag apache: $InputFileStateFile stat-apache2-error1 $InputFileSeverity error $InputFileFacility local7 $InputRunFileMonitor $InputFileName /var/log/apache2/ssl_request_log $InputFileTag apache: $InputFileStateFile stat-apache2-ssl1 $InputFileSeverity info $InputFileFacility local7 $InputRunFileMonitor
24
vRealize Automation, continued… # vRCS $InputFileName /storage/artifactory/home/logs/artifactory.log $InputFileTag vrcs: $InputFileStateFile stat-vrcs-artifactory $InputFileSeverity info $InputFileFacility local7 $InputRunFileMonitor $InputFileName /storage/artifactory/home/logs/import.export.log $InputFileTag vrcs: $InputFileStateFile stat-vrcs-import-export $InputFileSeverity info $InputFileFacility local7 $InputRunFileMonitor $InputFileName /storage/artifactory/home/logs/access_log $InputFileTag vrcs: $InputFileStateFile stat-vrcs-access1 $InputFileSeverity info $InputFileFacility local7 $InputRunFileMonitor $InputFileName /storage/artifactory/home/logs/error_log $InputFileTag vrcs: $InputFileStateFile stat-vrcs-error1 $InputFileSeverity error $InputFileFacility local7 $InputRunFileMonitor # check for new lines every 10 seconds $InputFilePollInterval 10 *.* @@<HOST>
25
Application Services • Log Insight agent configuration (recommended – copy and paste): ;;; vRA APPD [filelog|vra-appd] directory=/home/darwin/tcserver/darwin/logs event_marker=^\w+\s\d{2}\s\d{4}\s\S+\s\w+\s+[\S+] tags={"vmw_product":"vra","vmw_product_component":"appd"}
• Syslog configuration (restart syslog after changes): – /etc/syslog-ng/syslog-ng.conf – Add following details (substitute <HOST> with Syslog or Log Insight host at
end): # # APPD log files # Add to: /etc/syslog-ng/syslog-ng.conf # Replace with Log Insight FQDN # Run: /etc/init.d/syslog restart # source appd { file("/home/darwin/tcserver/darwin/logs/catalina.out" follow_freq(1) flags(no-parse) log_prefix("appd: ")); }; destination logserver { tcp("<HOST>" port (514)); }; log { source(appd); destination(logserver); }; log { source(src); destination(logserver); };
26
vRealize Automation Infrastructure • Log Insight agent configuration (copy and paste the static section, but
be sure to make changes to the dynamic section on next page): ;;; Static vRA agent configuration ;;; Just copy and paste the below configuration [filelog|vra-agent-vsphere] directory=C:\Program Files (x86)\VMware\vCAC\Agents\vSphereAgent\logs\ event_marker=^\[\d{1,2}/\d{1,2}/\d{4} \d{1,2}:\d{1,2}:\d{1,2} tags={"vmw_product":"vra","vmw_product_component":"agent"} [filelog|vra-server] directory=C:\Program Files (x86)\VMware\vCAC\Server\Logs\ include=*All.log;Repository.log event_marker=^\[\w\w\w:\d{4}-\d{2}-\d{2} tags={"vmw_product":"vra","vmw_product_component":"server"} [filelog|vra-mm] directory=C:\Program Files (x86)\VMware\vCAC\Server\Model Manager Web\Logs\ include=*All.log;Repository.log event_marker=^\[\w\w\w:\d{4}-\d{2}-\d{2} tags={"vmw_product":"vra","vmw_product_component":"mm"} [filelog|vra-web] directory=C:\Program Files (x86)\VMware\vCAC\Server\Website\Logs\ include=*All.log;Repository.log event_marker=^\[\w\w\w:\d{4}-\d{2}-\d{2} tags={"vmw_product":"vra","vmw_product_component":"web"} [filelog|vra-install] directory=C:\Program Files (x86)\VMware\vCAC\InstallLogs\ include=*All.log;Repository.log event_marker=^\[\w\w\w:\d{4}-\d{2}-\d{2} tags={"vmw_product":"vra","vmw_product_component":"install"}
27
vRealize Automation Infrastructure, continued… • Log Insight agent configuration, continued… ;;; Dynamic vRA agent configuration ;;; MANUAL CONFIGURATION CHANGES REQUIRED ;;; DO NOT JUST COPY AND PASTE THIS SECTION ;;; For every agent installed a new agent configuration section is required ;;; The name of the agent given during installation dictates the log directory name [filelog|vra-agent-<AGENT_NAME>] directory=C:\Program Files (x86)\VMware\vCAC\Agents\<AGENT_NAME>\logs\ event_marker=^\[\d{1,2}/\d{1,2}/\d{4} \d{1,2}:\d{1,2}:\d{1,2} tags={"vmw_product":"vra","vmw_product_component":"agent"} ;;; A DEM name can be specified during installation ;;; The name of the DEM given during installation dictates the log directory name ;;; If no name is given the DEM name is: DEM [filelog|vra-dem] directory=C:\Program Files (x86)\VMware\vCAC\Distributed Execution Manager\<DEM_NAME>\Logs\ include=*All.log;Repository.log event_marker=^\[\w\w\w:\d{4}-\d{2}-\d{2} tags={"vmw_product":"vra","vmw_product_component":"dem"} ;;; A DEO name can be specified during installation ;;; The name of the DEO given during installation dictates the log directory name ;;; If no name is given the DEO name is: DEO [filelog|vra-deo] directory=C:\Program Files (x86)\VMware\vCAC\Distributed Execution Manager\<DEO_NAME>\Logs\ include=*All.log;Repository.log event_marker=^\[\w\w\w:\d{4}-\d{2}-\d{2} tags={"vmw_product":"vra","vmw_product_component":"deo"}
28
vRealize Business Standard • Log Insight Agent configuration (copy and paste): ;;; vRB
[filelog|vra-vrb-server] directory=/var/log/itbm-server event_marker=^\d tags={"vmw_product":"vrb","vmw_product_component":"server"}
[filelog|vra-vrb-data-collector] directory=/var/log/itbm-data-collector event_marker=^\d tags={"vmw_product":"vrb","vmw_product_component":"data-collector"}
29
Log Insight
Aggregated Logs in Log Insight
31
Content Pack for vCAC 6.0 and vRA 6.1 or newer available on VMware Solution Exchange and the Log Insight marketplace
Related Documents
