The University of Akron Summit College Business Technology Dept.

The University of AkronSummit College

Business Technology Dept.2440: 141

Web Site Administration Web Server Monitoring and Analysis

Instructor: Enoch E. Damson

Monitoring and Analyzing SystemsMonitoring operating systems, Web servers,

applications, etc typically involves analyzing log filesLog files – contain information recorded by

the operating system in response to certain events

Monitoring and Analyzing the Web Server Environment 2

Monitoring Operating SystemsLogs are used to detect problems

OS, application, or security problemsVarious tools can monitor performance

Monitoring and Analyzing the Web Server Environment 3

Monitoring WindowsPerformance monitoring allows you to

compare system performance over timeWindows Task Manager highlights CPU and

memory usageYou can modify services to notify you if a

service fails

Monitoring and Analyzing the Web Server Environment 4

Windows Event Viewer The event viewer contains six event types shown in the left

pane

Monitoring and Analyzing the Web Server Environment 5

Windows Event LogsSystem and application events display

three levels of messagesInformationWarningError

Because many messages can be generated, a filter focuses on what you want to see

Over time, the logs fill up so you should clear them or save them

Monitoring and Analyzing the Web Server Environment 6

Monitoring LinuxLogging is controlled by the syslogd daemonBelow are some facilities which represent

daemons using syslogd

Monitoring and Analyzing the Web Server Environment 7

Eight Levels of Message Priorities in syslogd

Monitoring and Analyzing the Web Server Environment 8

Web Server Log FilesFiles that keep track of Web server

transactionsMost Web servers write two log files to

disk:Access log – contains a line for each Web

server requestError log – contains a line for each generated

error responseWhen log files grow:

A common practice is to put the log files on a separate drive or partition

A better solution is to rotate the log files Rename or remove the log files at regular intervals

(weekly, monthly, etc)Monitoring and Analyzing the Web Server Environment 9

Web Server Log File FormatsMost Web servers support at least two logging

formats:Common Logfile Format (CLF)Extended Logfile Format (ELF)

Most Web servers also allow the administrator to specify a custom format, along with the above formats

A standard logfile format makes it easier for users to understand files from different serversAllows third-party logfile analysis tools to support

many different Web servers

Monitoring and Analyzing the Web Server Environment 10

Common Logfile Format (CLF)The NCSA and CERN Web servers first used this

file formatMany Web servers now support this format (IIS,

Apache, Netscape Enterprise, etc)Each line in the file represents a unique requestHas a fixed format with seven fields to be logged:

remotehostrfc1413authuser [date]“request”statusbytes

Monitoring and Analyzing the Web Server Environment 11

Common Logfile Format… remotehost – remote (client) hostname or IP number rfc1413 – remote username

rfc1413 defines a protocol used to determine the identity of a client that requests a resource from the server

Seldom used on Internet servers because it slows the server’s response A “-” is entered into the log if the server is unable to determine a

userid authuser – when required, the username by which the user has

authenticated is provided A “-” is used for normal unrestricted requests

[date] – date and time of the request Enclosed in brackets for potential spaces

“request” – HTTP request line exactly as it came from the client Enclosed in quotes for potential spaces

status – HTTP status code returned to the client bytes – content length of document transferred Example:

127.0.0.1 - - [24/Oct/2006:09:11:55 -0500] "GET /test.asp HTTP/1.1" 200 626Monitoring and Analyzing the Web Server

Environment 12

Extended Logfile Format (ELF)Used to log more information or omit

certain fieldsAllows the administrator to specify exactly

which fields to log and in what orderEach represents a request like CLFs but

the beginning of the file also contains some configuration directivesEach directive line begins with a #Two directives are required and must precede

all entries in the log file: Version – specifies the version of the ELF to use Fields – specifies what data to record in the logfile

Monitoring and Analyzing the Web Server Environment 13

Extended Logfile Format…Example:

#Software: Microsoft Internet Information Services 5.1#Version: 1.0#Date: 2006-10-27 03:04:57#Fields: date time c-ip cs-method cs-uri-stem sc-status sc-bytes cs-version 2006-10-27 03:04:57 127.0.0.1 GET /test.asp 200 626 HTTP/1.1

The fields directive here specifies 8 out of several available fields: date – client request date time – client request time c-ip – client IP address cs-method – HTTP request method cs-uri-stem – file requested by client sc-status – HTTP status code returned to the client sc-bytes – number of bytes sent from server to client cs-version – version of HTTP used by client to connect to the server

Monitoring and Analyzing the Web Server Environment 14

Error LogsContains informational messages and debugging

informationUseful for:

Finding problems with the serverDebugging server-side programs and new configurations

Most server packages allow the administrator to control what types of messages are logged to the error log fileThe format is usually not configurable like ELFs but

allows some flexibility in choosing the severity and type of messages to log

E.g only critical messages may be logged if a server is running smoothly

Monitoring and Analyzing the Web Server Environment 15

ReferrersDetermines what Web page was used by

the client to access a serverMay be the URL of a search engine or any Web

site with a link to the Web serverA “-” is used if there was no Referrer header

sentThe Referrer header is not sent in the following

circumstances: The users enters the URL by hand The user clicked on a link to regular file and not a Web

page on a public site The user loaded the URL from a bookmark file The Referrer URL is on a private (internal) Web site The user or browser has disabled sending the Referrer

header Monitoring and Analyzing the Web Server Environment 16

Monitoring IISIIS has specific counters for use in the

Performance MonitorThe System event viewer provides specific

information IIS has extensive logging capabilities

There are default log formats used by various third-party applications that analyze logs

Monitoring and Analyzing the Web Server Environment 17

Monitoring ApacheError LogsBy default, syslogd sends Apache messages

to /var/log/boot.logLocation of the error log

ErrorLog logs/error_logLogs refers to /var/log/httpd

You can create a different error log for each virtual host

Monitoring and Analyzing the Web Server Environment 18

Monitoring ApacheTransfer LogsTransfer logs tell you about the use of

your Web siteThe default log is based on a combined

formatDetermined by the CustomLog directive in the

configuration file (httpd.conf)There are a number of sample formats

By default, logs are stored in /var/log/httpd/access_log

Monitoring and Analyzing the Web Server Environment 19

Monitoring DNSBIND uses a logging statement that you

configure in named.confBIND defines logging in two parts:

Channel defines where logging is sentCategory defines what will be sent

If the channel is going to a file, use the versions option to define the number of backupsSize option sets maximum size of the fileprint-time adds the date and time to the file

Monitoring and Analyzing the Web Server Environment 20

BIND Categories

Monitoring and Analyzing the Web Server Environment 21

Monitoring Exchange ServerExchange server uses the application

portion of Event viewerYou can enable four types of logs

audit – access to mailboxes protocol – commands used for SMTP, etcmessage tracking – senders and receiversdiagnostic – analyze detailed problems

Monitoring and Analyzing the Web Server Environment 22

Analysis Tools for the Web ServerAnalysis tools extract system data from

logs and format the dataFor IIS, one of the popular tools is

WebTrendsHelps you determine the source of Web

trafficDetermines which pages are most popularSeveral different reports

123LogAnalyzer is available for both IIS and ApacheMany reports are similar to WebTrends

Monitoring and Analyzing the Web Server Environment 23

Log File AnalysisSimply looking at log files can provide a lot of

information about activities or requests on a serverSimply counting the number of lines in an access

log file can help determine the number of hitsLog files may be reviewed regularly to find the

common errors loggedSome of the common errors include:

Dead links Requests for non-existing files CGI scripts not working properly Permissions problems

Some of the open-source log analyzers are: Analog (http://www.analog.cx) Webalizer (http://www.mrunnix.net/webalizer) Report Magic (http://www.reportmagic.org)

Monitoring and Analyzing the Web Server Environment 24

Statistics With the help of several log analyzer programs, some of the

statistical information that can be extracted include: Most requested pages Top entry pages (the first page clients enter a site through) Most used browsers Bandwidth utilization Most active domains Top referring sites and URLs Error counts Information about search engines (most common search engines,

common queries, etc) Some of the widely used commercial log analyzer products

include: WebTrends (http://www.webtrends.com) Wusage (http://www.boutell.com/wusage)

A database could also be used to store log information to increase efficiency of logging and report generation Not all Web servers support logging to a database

Monitoring and Analyzing the Web Server Environment 25