VMware vRealize Automation 6.1/6 · introduced in 6.0.1, ... NOTE: see the vRA Reference Architecture guide for detailed layouts. ... • Windows VIM on vCenter SSO

© 2014 VMware Inc. All rights reserved.

VMware vRealize Automation 6.1/6.2 Logging Overview and Configuration for Log Insight Content Pack

Kimberly Delgado, @KCDAutomate Steve Flanders, @smflanders

April 2015

Updates •  Content Pack

–  April 2015 – v1.1 includes updates to vRO queries to account for new log format introduced in 6.0.1, and update to General Overview, vRA events over time

•  Document –  April 2015 – updates to Agent doco links and instructions, and updates to vRO Log

config

2

Distributed Architecture

3

SSO

vRA VAs

vRO

Infrastructure Web / Manager LBs

Infrastructure Web / Manager Servers

Infrastructure Agents/DEMs

Clustered SQL Server DB

Infrastructure Fabric

NOTE: see the vRA Reference Architecture guide for detailed layouts.

vRA App Svcs

Host Roles •  SSO: Authentication from vCenter 5.5 SSO, SSO Identity Appliance or the

SSO standalone Windows host

•  vRA Virtual Appliance(s): host the CAFÉ & Code Stream services, embedded Postgres DB instance and embedded vRO Instance; CAFÉ services can be configured in distributed manner on multiple instances of the VA

•  vRO (External): for non-POC environments, an external vRO configuration can be standalone or load-balanced with external DB

•  App Services: VA for application services components

•  Infrastructure Web Server: hosts web server UI, WAPI interface and Model Manager

•  Manager Service: responsible for moving Infrastructure components through their defined lifecycle

•  DEMs: Orchestrator & Workers – interacts with Fabric sources

•  Agents – interacts with Fabric sources

4

Remote Logging •  Operating Systems

–  Windows does not natively support syslog –  Virtual Appliances (VAs) and Linux do support syslog

•  Log Insight Agent –  Available for both Windows and Linux –  Easy to deploy and configure; very lightweight –  Ability to handle multiline messages and tag events –  Properly handles log spikes and log rotation –  Offers capabilities beyond those provided by syslog

•  Use of the Log Insight agent is recommended for all vRealize Automation components (Windows and Linux) except vRO

•  The Log Insight agent configurations include custom Tags which are leveraged in the vRA content pack. If not properly configured, some queries may not work as expected.

5

Remote Logging, continued… •  The Log Insight agents can be downloaded directly from Log Insight,

under Administration / Management / Agents and follow the download link at the bottom of the page.

•  Log Insight Windows agent installation instructions:

http://pubs.vmware.com/log-insight-25/topic/com.vmware.log-insight.administration.doc/GUID-455106F4-4C3D-47C1-8EF6-84992BCCEB05.html

•  Log Insight Linux agent installation instructions (for the vRealize virtual appliances, choose RPM):

http://pubs.vmware.com/log-insight-25/topic/com.vmware.log-insight.administration.doc/GUID-DB4A27CF-BDA7-443F-94FB-AB9097AD8008.html

6

vRA 6.1+ Content Pack If you haven’t done so already, you can download the vRealize Automation 6.1+ Content Pack in one of two ways:

•  Logged into Log Insight as an Administrator, navigate to Content Packs / Marketplace. The VMware - vRA 6.1+ content pack will be near the bottom of the page.

•  Go to the VMware Solutions Exchange (https://solutionexchange.vmware.com/) and navigate to Cloud Management Marketplace, VMware vRealize Log Insight. You will need to scroll to locate or search for “vRA”.

7

Log Locations

vCenter SSO •  Identity VA (SSO VA & vCenter SSO)

–  /var/log/vmware/sso/* •  Catalina.out (primary) •  ssoAdminServer.log – user log in info here •  vmware-identity-sts-perf.log •  vmware-identity-sts.log •  vmware-sts-idmd-perf.log •  vmware-sts-idmd.err •  vmware-sts-idmd.log

–  /var/log/messages – Active Directory connection info

•  Windows VIM on vCenter SSO –  C:\ProgramData\VMware\CIS\logs\vmware-sso\ –  C:\ProgramData\VMware\CIS\runtime\VMwareSTS\logs

9

vRealize Automation – Virtual Appliances •  vRealize Automation (vRA) & Code Stream (vRCS)

–  /var/log/vmware/vcac/catalina.out

–  /var/log/apache2/access_log

–  /var/log/apache2/ssl_request_log

–  /var/log/apache2/error_log

•  vRealize Orchestrator (embedded & external same location): –  /var/log/vco/app-server/catalina.out

–  /var/log/vco/app-server/server.log

–  /var/log/vco/app-server/scripting.log

–  Individual plugins need to be configured for logging and may have different log locations

•  Application Services –  /home/darwin/tcserver/darwin/logs/catalina.out

•  Artifactory (for Code Stream) –  /storage/artifactory/home/logs/artifactory.log

–  /storage/artifactory/home/logs/access.log

–  /storage/artifactory/home/logs/request.log

–  /storage/artifactory/home/logs/import.export.log

10

vRealize Automation – Infrastructure •  Exact logs & locations will depend on deployment type and

configuration; these are basic places to start!

•  Infrastructure Server (Web, Manager) –  C:\Program Files (x86)\VMware\vCAC\Server\Logs\All –  C:\Program Files (x86)\VMware\vCAC\Server\Config Tool\Log\vCACConfiguration-

<date> –  C:\Program Files (x86)\VMware\vCAC\Server\Model Manager Data\Logs\ –  C:\Program Files (x86)\VMware\vCAC\Server\Model Manager Web\Logs\Repository –  C:\Program Files (x86)\VMware\vCAC\Website\Logs\Web_Admin_All –  C:\Program Files (x86)\VMware\vCAC\Web API\Logs\

11

vRealize Automation – Infrastructure, continued… •  Some log directories and filenames are set during installation and will

depend on entered information. Information like <THIS> needs to be replaced with entered information.

•  Agents –  C:\Program Files (x86)\VMware\vCAC\Agents\<PLUGIN>\logs\<FILE>

•  <PLUGIN> Examples: vSphereAgent, nsx, VC55Agent, VDIAgent •  <FILE> Examples: vSphereAgent, EpiPowerShellAgent, VdiPowerShellAgent IMPORTANT: The Agent name specified during installation dictates the value of <PLUGIN>

•  DEMs –  C:\Program Files (x86)\VMware\vCAC\Distributed Execution Manager\<DEM_NAME> –  C:\Program Files (x86)\VMware\vCAC\Distributed Execution Manager\<DEO_NAME> –  IMPORTANT: The DEM/DEO name specified during installation dictates the value

12

vRealize Business •  vRB Data Collector

–  /var/log/itbm-data-collector/catalina.out –  /var/log/itbm-data-collector/itfm-vc-dc.log –  /var/log/itbm-data-collector/localhost_access_log.* –  /var/log/itbm-data-collector/vf.tc-events.txt

•  vRB Server –  /var/log/itbm-server/audit.log –  /var/log/itbm-server/catalina.out –  /var/log/itbm-server/itfm-external-api.log –  /var/log/itbm-server/itfm-reflib-update.log –  /var/log/itbm-server/itfm.log –  /var/log/itbm-server/localhost_access_log.* –  /var/log/itbm-server/vcac.log –  /var/log/itbm-server/vf.tc-events.txt

13

Syslog Configuration

Log Insight Server-Side Agent Configuration •  Log Insight agent configuration can be set client-side or server-side.

Server-side consists of three steps outlined below. The slide following have client-side configurations.

1.  Enable vRO logging – see the vRO slide for configuration information

2.  Static configuration (copy and paste): ;;; vCenter SSO VCSA [filelog|vmw-sso] directory=/var/log/vmware/sso exclude=vmware-* event_marker=^(\[\d{4}-\d{2}-\d{2}|\d{2}-\w+-\d{4}) tags={"vmw_product":"sso"}

[filelog|vmw-sso-sts-idmd-perf] directory=/var/log/vmware/sso include=vmware-sts-idmd-perf* event_marker=^\d{4}-\d{2}-\d{2}\s\S+\s\w+\s+\w+ tags={"vmw_product":"sso"}

[filelog|vmw-sso-sts-perf] directory=/var/log/vmware/sso include=vmware-identity-sts-perf* event_marker=^\[\d{4}-\d{2}-\d{2}\s\S+\s\S+\s\S+\]\s+\w+ tags={"vmw_product":"sso"}

[filelog|vmw-sso-sts-other] directory=/var/log/vmware/sso include=vmware-sts-idmd.*;vmware-identity-sts.* event_marker=^\[\d{4}-\d{2}-\d{2}\s\S+\s\S+\s\S+ tags={"vmw_product":"sso"}

15

Log Insight Server-Side Agent Configuration ;;; vCenter SSO Windows [filelog|vcenter-sso] directory=C:\ProgramData\VMware\CIS\logs\vmware-sso event_marker=^\[\d{1,2}/\d{1,2}/\d{4} \d{1,2}:\d{1,2}:\d{1,2} tags={"vmw_product":"sso"} [filelog|vcenter-sso-sts] directory=C:\ProgramData\VMware\CIS\runtime\VMwareSTS\logs event_marker=^\[\d{1,2}/\d{1,2}/\d{4} \d{1,2}:\d{1,2}:\d{1,2} tags={"vmw_product":"sso"}

;;; vRA

[filelog|vra] directory=/var/log/vmware/vcac event_marker=^\d tags={"vmw_product":"vra","vmw_product_component":"cafe"}

[filelog|apache] directory=/var/log/apache2 tags={"asf_product":"http"}

;;; vRCS

[filelog|vrcs] directory=/storage/artifactory/home/logs event_marker=^\d tags={"vmw_product":"vrcs","vmw_product_component":"jfrog"}

;;; vRA APPD [filelog|vra-appd] directory=/home/darwin/tcserver/darwin/logs event_marker=^\w+\s\d{2}\s\d{4}\s\S+\s\w+\s+[\S+] tags={"vmw_product":"vra","vmw_product_component":"appd"}

16

Log Insight Server-Side Agent Configuration ;;; Static vRA [filelog|vra-agent-vsphere] directory=C:\Program Files (x86)\VMware\vCAC\Agents\vSphereAgent\logs\ event_marker=^\[\d{1,2}/\d{1,2}/\d{4} \d{1,2}:\d{1,2}:\d{1,2} tags={"vmw_product":"vra","vmw_product_component":"agent"} [filelog|vra-server] directory=C:\Program Files (x86)\VMware\vCAC\Server\Logs\ include=*All.log;Repository.log event_marker=^\[\w\w\w:\d{4}-\d{2}-\d{2} tags={"vmw_product":"vra","vmw_product_component":"server"} [filelog|vra-mm] directory=C:\Program Files (x86)\VMware\vCAC\Server\Model Manager Web\Logs\ include=*All.log;Repository.log event_marker=^\[\w\w\w:\d{4}-\d{2}-\d{2} tags={"vmw_product":"vra","vmw_product_component":"mm"} [filelog|vra-web] directory=C:\Program Files (x86)\VMware\vCAC\Server\Website\Logs\ include=*All.log;Repository.log event_marker=^\[\w\w\w:\d{4}-\d{2}-\d{2} tags={"vmw_product":"vra","vmw_product_component":"web"} [filelog|vra-install] directory=C:\Program Files (x86)\VMware\vCAC\InstallLogs\ include=*All.log;Repository.log event_marker=^\[\w\w\w:\d{4}-\d{2}-\d{2} tags={"vmw_product":"vra","vmw_product_component":"install"}

17

Log Insight Server-Side Agent Configuration ;;; vRB

[filelog|vra-vrb-server] directory=/var/log/itbm-server event_marker=^\d tags={"vmw_product":"vrb","vmw_product_component":"server"}

[filelog|vra-vrb-data-collector] directory=/var/log/itbm-data-collector event_marker=^\d tags={"vmw_product":"vrb","vmw_product_component":"data-collector"}

3.  Dynamic configuration (modify everything like <THIS>) because folder names are set based on install parameters provided:

;;; Dynamic vRA agent configuration ;;; MANUAL CONFIGURATION CHANGES REQUIRED ;;; DO NOT JUST COPY AND PASTE THIS SECTION ;;; For every agent installed a new agent configuration section is required ;;; The name of the agent given during installation dictates the log directory name [filelog|vra-agent-<AGENT_NAME>] directory=C:\Program Files (x86)\VMware\vCAC\Agents\<AGENT_NAME>\logs\ event_marker=^\[\d{1,2}/\d{1,2}/\d{4} \d{1,2}:\d{1,2}:\d{1,2} tags={"vmw_product":"vra","vmw_product_component":"agent"}

18

Log Insight Server-Side Agent Configuration ;;; A DEM name can be specified during installation ;;; The name of the DEM given during installation dictates the log directory name ;;; If no name is given the DEM name is: DEM [filelog|vra-dem] directory=C:\Program Files (x86)\VMware\vCAC\Distributed Execution Manager\<DEM_NAME>\Logs\ include=*All.log;Repository.log event_marker=^\[\w\w\w:\d{4}-\d{2}-\d{2} tags={"vmw_product":"vra","vmw_product_component":"dem"} ;;; A DEO name can be specified during installation ;;; The name of the DEO given during installation dictates the log directory name ;;; If no name is given the DEO name is: DEO [filelog|vra-deo] directory=C:\Program Files (x86)\VMware\vCAC\Distributed Execution Manager\<DEO_NAME>\Logs\ include=*All.log;Repository.log event_marker=^\[\w\w\w:\d{4}-\d{2}-\d{2} tags={"vmw_product":"vra","vmw_product_component":"deo"}

19

vCenter SSO on VCSA •  Log Insight agent configuration (recommended – copy and paste): ;;; vCenter SSO VCSA [filelog|vmw-sso] directory=/var/log/vmware/sso exclude=vmware-* event_marker=^(\[\d{4}-\d{2}-\d{2}|\d{2}-\w+-\d{4}) tags={"vmw_product":"sso"}

[filelog|vmw-sso-sts-idmd-perf] directory=/var/log/vmware/sso include=vmware-sts-idmd-perf* event_marker=^\d{4}-\d{2}-\d{2}\s\S+\s\w+\s+\w+ tags={"vmw_product":"sso"}

[filelog|vmw-sso-sts-perf] directory=/var/log/vmware/sso include=vmware-identity-sts-perf* event_marker=^\[\d{4}-\d{2}-\d{2}\s\S+\s\S+\s\S+\]\s+\w+ tags={"vmw_product":"sso"}

[filelog|vmw-sso-sts-other] directory=/var/log/vmware/sso include=vmware-sts-idmd.*;vmware-identity-sts.* event_marker=^\[\d{4}-\d{2}-\d{2}\s\S+\s\S+\s\S+ tags={"vmw_product":"sso"}

•  Syslog configuration (restart syslog after changes): –  /etc/syslog-ng/syslog-ng.conf –  Set "destination logserver" to syslog host or Log Insight

20

vCenter SSO on Windows •  Log Insight agent configuration (copy and paste): ;;; vCenter SSO Windows [filelog|vcenter-sso] directory=C:\ProgramData\VMware\CIS\logs\vmware-sso event_marker=^\[\d{1,2}/\d{1,2}/\d{4} \d{1,2}:\d{1,2}:\d{1,2} tags={"vmw_product":"sso"} [filelog|vcenter-sso-sts] directory=C:\ProgramData\VMware\CIS\runtime\VMwareSTS\logs event_marker=^\[\d{1,2}/\d{1,2}/\d{4} \d{1,2}:\d{1,2}:\d{1,2} tags={"vmw_product":"sso"}

21

vRealize Orchestrator (vRO) •  Syslog configuration: Edit /etc/vco/app-server/log4j.xml

–  If Syslog was configured in VAMI, you MUST change the ”vco: prio:" section below highlighted in red to match how it is shown here, or you will get no vRO logs from the content pack queries!

–  Edit section (remove comments and substitute <HOST> with Syslog or Log Insight host):

<appender name="SYSLOG" class="org.apache.log4j.net.SyslogAppender"> <param name="Threshold" value="INFO"/> <param name="Facility" value="LOCAL1"/> <param name="SyslogHost" value="<HOST>"/> <param name="FacilityPrinting" value="false"/> <layout class="org.apache.log4j.PatternLayout"> <param name="ConversionPattern" value="vco: prio:%-5p thread:%t token:%X{token} wf:%X{workflowName} wfid:%X{workflow} user: %X{username} cat:%c{1} msg:%m%n"/> </layout> </appender>

•  At end of config xml (/etc/vco/app-server/log4j.xml) –  Edit section (remove comments for SYSLOG appender):

<root> <priority value="INFO" /> <appender-ref ref="CONSOLE" /> <appender-ref ref="FILE" /> <appender-ref ref="SYSLOG" /> <!-- <appender-ref ref="EVENT_LOG" /> --> </root>

22

vRealize Automation (vRA) & Code Stream (vRCS) •  Log Insight agent configuration (recommended – copy and paste): ;;; vRA

[filelog|vra] directory=/var/log/vmware/vcac event_marker=^\d tags={"vmw_product":"vra","vmw_product_component":"cafe"}

[filelog|apache] directory=/var/log/apache2 tags={"asf_product":"http"}

;;; vRCS

[filelog|vrcs] directory=/storage/artifactory/home/logs event_marker=^\d tags={"vmw_product":"vrcs","vmw_product_component":"jfrog"}

•  Syslog configuration (restart syslog after changes):

–  /etc/rsyslog.d/remote.conf –  Add details for each log file (substitute <HOST> with Syslog or Log Insight

host at end):

23

vRealize Automation, continued… # # vRA + vRCS log files # Add to: /etc/rsyslog.d/remote.conf # Replace with Log Insight FQDN # Run: /etc/init.d/syslog restart # $ModLoad imfile # vRA $InputFileName /var/log/vmware/vcac/catalina.out $InputFileTag vcac: $InputFileStateFile stat-vcac-catalina1 $InputFileSeverity info $InputFileFacility local7 $InputRunFileMonitor $InputFileName /var/log/vco/app-server/catalina.out $InputFileTag vco: $InputFileStateFile stat-vco-catalina1 $InputFileSeverity info $InputFileFacility local7 $InputRunFileMonitor $InputFileName /var/log/apache2/access_log $InputFileTag apache: $InputFileStateFile stat-apache2-access1 $InputFileSeverity info $InputFileFacility local7 $InputRunFileMonitor $InputFileName /var/log/apache2/error_log $InputFileTag apache: $InputFileStateFile stat-apache2-error1 $InputFileSeverity error $InputFileFacility local7 $InputRunFileMonitor $InputFileName /var/log/apache2/ssl_request_log $InputFileTag apache: $InputFileStateFile stat-apache2-ssl1 $InputFileSeverity info $InputFileFacility local7 $InputRunFileMonitor

24

vRealize Automation, continued… # vRCS $InputFileName /storage/artifactory/home/logs/artifactory.log $InputFileTag vrcs: $InputFileStateFile stat-vrcs-artifactory $InputFileSeverity info $InputFileFacility local7 $InputRunFileMonitor $InputFileName /storage/artifactory/home/logs/import.export.log $InputFileTag vrcs: $InputFileStateFile stat-vrcs-import-export $InputFileSeverity info $InputFileFacility local7 $InputRunFileMonitor $InputFileName /storage/artifactory/home/logs/access_log $InputFileTag vrcs: $InputFileStateFile stat-vrcs-access1 $InputFileSeverity info $InputFileFacility local7 $InputRunFileMonitor $InputFileName /storage/artifactory/home/logs/error_log $InputFileTag vrcs: $InputFileStateFile stat-vrcs-error1 $InputFileSeverity error $InputFileFacility local7 $InputRunFileMonitor # check for new lines every 10 seconds $InputFilePollInterval 10 *.* @@<HOST>

25

Application Services •  Log Insight agent configuration (recommended – copy and paste): ;;; vRA APPD [filelog|vra-appd] directory=/home/darwin/tcserver/darwin/logs event_marker=^\w+\s\d{2}\s\d{4}\s\S+\s\w+\s+[\S+] tags={"vmw_product":"vra","vmw_product_component":"appd"}

•  Syslog configuration (restart syslog after changes): –  /etc/syslog-ng/syslog-ng.conf –  Add following details (substitute <HOST> with Syslog or Log Insight host at

end): # # APPD log files # Add to: /etc/syslog-ng/syslog-ng.conf # Replace with Log Insight FQDN # Run: /etc/init.d/syslog restart # source appd { file("/home/darwin/tcserver/darwin/logs/catalina.out" follow_freq(1) flags(no-parse) log_prefix("appd: ")); }; destination logserver { tcp("<HOST>" port (514)); }; log { source(appd); destination(logserver); }; log { source(src); destination(logserver); };

26

vRealize Automation Infrastructure •  Log Insight agent configuration (copy and paste the static section, but

be sure to make changes to the dynamic section on next page): ;;; Static vRA agent configuration ;;; Just copy and paste the below configuration [filelog|vra-agent-vsphere] directory=C:\Program Files (x86)\VMware\vCAC\Agents\vSphereAgent\logs\ event_marker=^\[\d{1,2}/\d{1,2}/\d{4} \d{1,2}:\d{1,2}:\d{1,2} tags={"vmw_product":"vra","vmw_product_component":"agent"} [filelog|vra-server] directory=C:\Program Files (x86)\VMware\vCAC\Server\Logs\ include=*All.log;Repository.log event_marker=^\[\w\w\w:\d{4}-\d{2}-\d{2} tags={"vmw_product":"vra","vmw_product_component":"server"} [filelog|vra-mm] directory=C:\Program Files (x86)\VMware\vCAC\Server\Model Manager Web\Logs\ include=*All.log;Repository.log event_marker=^\[\w\w\w:\d{4}-\d{2}-\d{2} tags={"vmw_product":"vra","vmw_product_component":"mm"} [filelog|vra-web] directory=C:\Program Files (x86)\VMware\vCAC\Server\Website\Logs\ include=*All.log;Repository.log event_marker=^\[\w\w\w:\d{4}-\d{2}-\d{2} tags={"vmw_product":"vra","vmw_product_component":"web"} [filelog|vra-install] directory=C:\Program Files (x86)\VMware\vCAC\InstallLogs\ include=*All.log;Repository.log event_marker=^\[\w\w\w:\d{4}-\d{2}-\d{2} tags={"vmw_product":"vra","vmw_product_component":"install"}

27

vRealize Automation Infrastructure, continued… •  Log Insight agent configuration, continued… ;;; Dynamic vRA agent configuration ;;; MANUAL CONFIGURATION CHANGES REQUIRED ;;; DO NOT JUST COPY AND PASTE THIS SECTION ;;; For every agent installed a new agent configuration section is required ;;; The name of the agent given during installation dictates the log directory name [filelog|vra-agent-<AGENT_NAME>] directory=C:\Program Files (x86)\VMware\vCAC\Agents\<AGENT_NAME>\logs\ event_marker=^\[\d{1,2}/\d{1,2}/\d{4} \d{1,2}:\d{1,2}:\d{1,2} tags={"vmw_product":"vra","vmw_product_component":"agent"} ;;; A DEM name can be specified during installation ;;; The name of the DEM given during installation dictates the log directory name ;;; If no name is given the DEM name is: DEM [filelog|vra-dem] directory=C:\Program Files (x86)\VMware\vCAC\Distributed Execution Manager\<DEM_NAME>\Logs\ include=*All.log;Repository.log event_marker=^\[\w\w\w:\d{4}-\d{2}-\d{2} tags={"vmw_product":"vra","vmw_product_component":"dem"} ;;; A DEO name can be specified during installation ;;; The name of the DEO given during installation dictates the log directory name ;;; If no name is given the DEO name is: DEO [filelog|vra-deo] directory=C:\Program Files (x86)\VMware\vCAC\Distributed Execution Manager\<DEO_NAME>\Logs\ include=*All.log;Repository.log event_marker=^\[\w\w\w:\d{4}-\d{2}-\d{2} tags={"vmw_product":"vra","vmw_product_component":"deo"}

28

vRealize Business Standard •  Log Insight Agent configuration (copy and paste): ;;; vRB

[filelog|vra-vrb-server] directory=/var/log/itbm-server event_marker=^\d tags={"vmw_product":"vrb","vmw_product_component":"server"}

[filelog|vra-vrb-data-collector] directory=/var/log/itbm-data-collector event_marker=^\d tags={"vmw_product":"vrb","vmw_product_component":"data-collector"}

29

Log Insight

Aggregated Logs in Log Insight

31

Content Pack for vCAC 6.0 and vRA 6.1 or newer available on VMware Solution Exchange and the Log Insight marketplace