Exchange Server 2007 SP1Core Roles Troubleshooting
João [email protected]
Agenda
Introduction to Exchange Server 2007 Troubleshooting
Troubleshooting Client Access Server (CAS)
Troubleshooting Hub Transport Server (HT)
Troubleshooting Mailbox Server (MBX)
Microsoft Confidential2
Introducing to Exchange Server 2007 Troubleshooting
Introduction
Exchange Services Overview
Active Directory Provider
Microsoft Confidential4
Exchange Services shared by all Server Roles
Microsoft Confidential5
Microsoft Exchange Active Directory Topology (MSExchangeADTopology)
Provides Active Directory topology information to several Exchange Server components
This service does not have any dependencies
EDGE not dependent
Microsoft Exchange Monitoring (MSExchangeMonitoring)
Provides a remote procedure call (RPC) server, used to invoke diagnostic cmdlets.
Microsoft Exchange Transport Log Search(MSExchangeTransportLogSearch)
Provides message tracking and transport log searching
UM not dependent
Exchange Client Access Role Services
Microsoft Confidential6
Microsoft Exchange File Distribution Service (MSExchangeFDS)
Used to distribute offline address book
Dependent upon AD Topology Service
Microsoft Exchange IMAP4 (MSExchangeIMAP4)
Provides IMAP4 services to IMAP clients
Dependent upon AD Topology Service
Microsoft Exchange POP3 (MSExchangePOP3)
Provides POP3 services to POP3 clients
Dependent upon AD Topology Service
Microsoft Exchange Service Host (MSExchangeServiceHost)
Config of RPC virtual directory in IIS
Registry Config or ValidPorts, NSPI Interface Protocol Sequences, and AllowAnonymous for Outlook Anywhere
Dependent upon AD Topology Service
Exchange Hub Transport Role Services
Microsoft Confidential7
Microsoft Exchange Transport (MSExchangeTransport)
Provides SMTP server and transport stack
Dependent upon AD Topology Service
Microsoft Exchange EdgeSync (MSExchangeEdgeSync)
Connects to ADAM instance on subscribed Edge Transport servers over secure LDAP
If there are no Edge Subscriptions configured, this service can be disabled
Dependent upon AD Topology Service
Microsoft Exchange Anti-spam Update (MSExchangeAntispamUpdate)
Auto download anti-spam filter updates
Exchange Edge Role Services
Microsoft Confidential8
Microsoft Exchange Transport (MSExchangeTransport)
Provides SMTP server and transport stack
Microsoft Exchange ADAM (ADAM_MSExchange)
Stores configuration and recipient data on the Edge Transport server
Microsoft Exchange Credential Service (EdgeCredentialSvc)
Monitors credential changes in ADAM and installs the changes on the Edge Transport server
Microsoft Exchange Anti-spam Update (MSExchangeAntispamUpdate)
Auto download anti-spam filter updates
Exchange Mailbox Role Services
Microsoft Confidential9
Microsoft Exchange Information Store (MSExchangeIS)
Manages Exchange Server databases
Provides data storage for messaging clients
Dependent upon : Event Log, NT LM Security Support Provider, Remote Procedure Call (RPC), Server, and Workstation
Microsoft Exchange System Attendant (MSExchangeSA)
Provides monitoring, maintenance, and directory lookup services
Dependent upon: Event Log, NT LM Security Support Provider, Remote Procedure Call (RPC), Server, and Workstation
Microsoft Exchange Mail Submission Service (MSExchangeMailSubmission)
Notifies Hub Transport server located in the Mailbox server's Active Directory to pickup from a sender's outbox
Dependent upon AD Topology Service
Exchange Mailbox Role Services cont
Microsoft Confidential10
Microsoft Exchange Mailbox Assistants(MSExchangeMailboxAssistants)
Provides functionality for Calendar Attendant, Resource Booking Attendant, Out of Office Assistant, and Managed Folder Mailbox Assistant.
Dependent upon AD Topology Service
Microsoft Exchange Replication Service (MSExchangeRepl)
Provides log shipping functionality for LCR, CCR and SCR.
Dependent upon AD Topology Service
Microsoft Exchange Service Host (MSExchangeServiceHost)
Config of RPC virtual directory in IIS
Registry Config or ValidPorts, NSPI Interface Protocol Sequences, and AllowAnonymous for Outlook Anywhere
Dependent upon AD Topology Service
Exchange Mailbox Role Services cont
Microsoft Confidential11
Microsoft Exchange Search Indexer (MSExchangeSearch)
Provides content to the Microsoft Search (Exchange Server) service for indexing.
Dependent upon AD Topology Service and the Microsoft Search (Exchange Server) service.
Microsoft Search (Exchange Server) (MSFTESQL-Exchange)
Provides full-text indexing of mailbox data content
Exchange-customized version of Microsoft Search
Dependent upon the Remote Procedure Call (RPC) service
Exchange Unified Messaging Role Services
Microsoft Confidential12
Microsoft Exchange File Distribution Service (MSExchangeFDS)
Distribute custom Unified Messaging prompts
Dependent upon AD Topology Service
Microsoft Exchange Speech Engine (MSS)
Provides speech processing services
Dependent upon Windows Management Instrumentation service
Microsoft Exchange Unified Messaging (MSExchangeUM)
Provides Unified Messaging features:
Storing inbound faxes and voice mail messages
Access to mailbox via Outlook Voice Access
Dependent upon AD Topology Service and the Microsoft Exchange Speech Engine service
Active Directory Provider
What is the AD Provider?
Components
Who uses the AD Provider?
Exchange Active Directory Topology Service
AD Topology vs Exchange Topology
Microsoft Confidential13
Active Directory Provider
The majority of components and services in Exchange 2007 are built on managed code:
Replay Service
Exchange Transport Service
Mailbox Assistants
Search
Unified Messaging
Unmanaged componentsInformation Store Service
System Attendant
DAV
Microsoft Confidential14
Active Directory Provider: What is the AD Provider?
What is the AD Provider?New component in Exchange 2007 that leverages the advantages of being built on managed code
Provides an efficient, robust mechanism for managed applications to communicate with Active Directory
Loaded by all managed applications
Unmanaged applications continue to load DSAccess.dll
Improvements over DSAccess include
Support for paged queries
Support for very large multivalued attributes
Does not access and store directory information in a cache
Microsoft Confidential15
Active Directory Provider: Components
AD Recipient Objects
AD System Configuration Objects
AD Driver
The engine inside the AD Provider
Provides the following functions:
Determines which server to connect to
Maintains connections pools
Performs connection failover
Manages Exchange and AD topology discovery
CRUD operations against AD
Provides interfaces to recipient and system configuration objects
Microsoft Confidential16
Active Directory Provider: Who uses the AD Provider?
Who uses the AD Provider?Microsoft Exchange EdgeSync
Microsoft Exchange File Distribution
Microsoft Exchange Service Host
Microsoft Exchange Transport
Microsoft Exchange Transport Log Search
Microsoft Exchange Replication Service
Microsoft Exchange Mail Submission
Microsoft Exchange Mailbox Assistants
Microsoft Exchange Search Indexer
Microsoft Exchange Unified Messaging
Exchange Management Console
Exchange Management Shell
Setup
Microsoft Confidential17
Active Directory Provider: Exchange AD Topology Service
Unmanaged Windows service that provides an RPC server interface in order to allow managed code processes access to AD topology information maintained by DSAccess
Effectively a wrapper for DSAccess that makes specific DSAccess functions available via RPC to the AD Driver running within managed code process
Dependency service for all managed applications
Microsoft Confidential18
Active Directory Provider: AD Provider and AD Topology Service
AD Provider and AD Topology Service
Microsoft Confidential19
Active Directory Provider: AD Topology vs Exchange Topology
Active Directory TopologyList of DCs and GCs in the local site and closest sites
Details about each server:
Is it reachable via ping?
Is it a DC or GC?
Is it synchronized?
Which domain it lives in?
OS version
SACL Right
This data is used in failover and load balancing
Microsoft Confidential20
Exchange TopologyAD sites, site links and costs
Subnets
VDirs
Location of Exchange servers
Examples of use:
Mail Routing
Mapping of Client Access, Hub Transport and Unified Messaging server to the appropriate Mailbox server
PF referrals
Troubleshooting Client Access Server (CAS)
Agenda
Introduction
Overview
Locating CAS Configuration and Topology Data
Troubleshooting:
Autodiscover
Availability Service
Offline Address Book
Client Access Security
Outlook Web Access
Exchange ActiveSync
Microsoft Confidential22
Introduction – What CAS is used for?
• AutoDiscover Service
• Availability Service
• Calendaring/Scheduling Assistant
• OOF configuration
• UM configuration
Outlook 2007 Clients only
• Outlook Anywhere
• Exchange ActiveSync
• POP3/IMAP4
• Outlook Web Access
Previous Version Clients
Microsoft Confidential23
Introduction – CAS Role
At least one in each AD site that contains Mailbox server roleIt is NOT supported to work with no CAS server
CAS role can be combined with any other role except:Edge Transport Server role
A server in a cluster
Microsoft Confidential25
Agenda
Introduction
Overview
Locating CAS Configuration and Topology Data
Troubleshooting:
Autodiscover
Availability Service
Offline Address Book
Client Access Security
Outlook Web Access
Exchange ActiveSync
Microsoft Confidential26
Client Access Server - Overview
Microsoft Confidential27
Site A
Site B
Outlook Express
Outlook Web AccessWindows Mobile
HTTPS
HTTPSIMAP4/POP3
SMTP
Encrypted RPC(MAPI)
Outlook 2007(via Outlook Anywhere)
Outlook 2003/2007(via MAPI)
Outlook
Anywhere
RPC
(MAPI)
Encrypted RPC(MAPI)
Encrypted RPC(MAPI)
Client Accessserver roles
Hub Transportserver roles Mailbox
server roles
Mailboxserver roles
Hub Transportserver roles
Client Accessserver roles
HTTPS
Encrypted RPC(MAPI)
SMTP over TLS
Agenda
Introduction
Overview
Locating CAS Configuration and Topology Data
Troubleshooting:
Autodiscover
Availability Service
Offline Address Book
Client Access Security
Outlook Web Access
Exchange ActiveSync
Microsoft Confidential29
Locating CAS Configuration and Topology data
Critical Data Location
Microsoft Office Outlook Web
Access Web Site and Web.config
file
File System \Client Access\Owa
IMAP4 and POP3 protocol settings File system \ClientAccess\PopImap
Availability service Active Directory configuration container and
file system, including the Web.config file
\ClientAccess\exchweb\ews
Autodiscover •IIS metabase
•Active Directory configuration container
Exchange ActiveSync •Active Directory configuration container
•File system, including the Web.config file in
the \ ClientAccess\Sync folder
•IIS Metabase
Outlook Web Access virtual
directories
Active Directory configuration container and
file system \ClientAccess\
Web services configuration IIS metabase
Microsoft Confidential30
Agenda
Introduction
Overview
Locating CAS Configuration and Topology Data
Troubleshooting:
Autodiscover
Availability Service
Offline Address Book
Client Access Security
Outlook Web Access
Exchange ActiveSync
Microsoft Confidential31
Autodiscover
What Autodiscover does:Automatically configure Outlook profiles without knowing where the mailbox is located
Provide Web Service URLs to Outlook 2007 Clients
Use both RPC and HTTPS connection
What Autodiscover doesn‘t configure:
Cache mode vs. Online mode
Security Settings
Remote Mail Settings
Anything under Tools/Options in Outlook
Microsoft Confidential32
Service Connection Point
CAS Installation will create:A new virtual directory called „AutoDiscover“ in IIS
Used by Outlook 2007 Clients only
Service Connection Point (SCP) in Active Directory which contains authoritative list of AutoDiscover Service URLs
CN=<CAS_server>,CN=AutoDiscover,CN=Protocols,CN=<CAS_Server>,CN=Servers,CN=Exchange Administrative Group,CN=AdministrativeGroup, CN=<Organization>,CN=Services,[Configuration Naming Context]
SCP Objects are accessed by domain joined Outlook 2007 clients to locate AutoDiscover service
Non domain joined clients rely on DNS to locate AutoDiscoverservice
Microsoft Confidential33
Autodiscover
Configuring Outlook 2007 profiles and Web services URLs
Microsoft Confidential34
Exchange
captures Outlook request, and
builds specific
connection settings for
Outlook AD
XML
Config
AutoDiscover
Service
HTTP
Request
Configuration
Information
AD
Lookup
autodiscover.contoso.com
Outlook 2007
1 Outlook uses e-mail address
to locate an Exchange Client Access servers at a
pre-defined location
(autodiscover.domain.com)
2
Configuration settings
are downloaded by Outlook and applied
to profile
3
Client
Access
server
role
0 If domain joined, Outlook
automatically fills out the user‟s email address and
password
•Outlook Anywhere
settings
• Server locations
• Web service URLs
• Authentication
information
• OAB download
location
Locating Autodiscover
To locate AutoDiscover:Internal domain joined clients use SCP
Non domain joined or external clients use DNS
For Outlook Anywhere and remote clients an host record for Autodiscover server should be created on an external DNS
Without AutoDiscover access client can access mailbox but certain functions like F/B, OOF, OAB and UM will not be accessible
If AutoDiscover is located via DNS, Outlook will try pre-determined order of URLs to connect to AutoDiscover Server. For example:
https://domain.com/autodiscover/autodiscover.xml
https://autodiscover.domain.com/autodiscover/autocover.xml
Microsoft Confidential36
New DNS SRV Record for Locating Autodiscover Service
Predefined URL method requires valid SSL certificate for URLs being used
Generally different DNS names are used for Outlook Anywhere and OWA
HTTP Redirect needs additional Web Site in IIS and two Public IP Addresses
New Software for Outlook performs and additional check forDNS SRV record for Autodiscover Service
This feature is available as part of following update rollup forOutlook : (http://support.microsoft.com/kb/939184/) Description of the update rollup for Outlook 2007: June 27, 2007
Microsoft Confidential38
Troubleshoot Autodiscover
Client side
Test E-mail Autoconfiguration
Outlook Logging
Server side
Test-OutlookWebServices
Event Logs
Exchange Management Shell
Microsoft Confidential40
Troubleshoot Autodiscover – Client.
Results tab: Web service URLs
Log tab: URLs used and error codes
Popular error codes
80072EE7 – ERROR_INTERNET_NAME_NOT_RESOLVED
80072EFD – ERROR_INTERNET_CANNOT_CONNECT
80072F17 – ERROR_INTERNET_SEC_CERT_ERRORS
Outlook logging
OLKDISC.log in temp directory
OLKAS directory
Microsoft Confidential41
Troubleshoot Autodiscover – Test-OutlookWebService
Examples of failures:
Microsoft Confidential42
When using
self signed
certificates
A DNS issue
or a general
server
performance
problem
The
AutoDiscover
XML file is
missing.
Troubleshoot Autodiscover – Server Event Logs
Event logging
Three MSExchange AutoDiscover event categories\Core
\Provider
\Web
Set-EventLogLevel "MSExchange AutoDiscover\Core" –Level:Expert
Microsoft Confidential43
Agenda
Introduction
Overview
Locating CAS Configuration and Topology Data
Troubleshooting:
Autodiscover
Availability Service
Offline Address Book
Client Access Security
Outlook Web Access
Exchange ActiveSync
Microsoft Confidential44
Availability Service
Calendaring functionality for free/busy, meeting suggestions and Out-of-Office (OOF) depends on Availability Web Service
Availability Service is used only by Exchange 2007 Mailboxes
For Exchange 2007 Mailboxes, Calendar data will be read from user‘s mailbox directly
OL 2007/Exchange 2007 users access Exchange 2003 mailbox free/busy data by using Availability Service to look up free/busy Public Folders on Exchange 2003 Servers
Microsoft Confidential45
Availability Service – Cross Forest Access
To access cross-forest free/busy data, make sure free/busy information is replicated between forests
To see free/busy data of Exchange 2003 mailboxes in the other forest, configure the availability service by running the following command on any server in the Exchange 2007 forest:
Add-AvailabilityAddressSpace -ForestName:<forest name e.g. msft.com> -AccessMethod:PublicFolder
Microsoft Confidential46
Troubleshooting Availability Service
Microsoft Confidential47
In Outlook 2007, Ctrl-Right-click on the Outlook system tray icon.
Enter your email address and password
Troubleshooting Availability Service, cont.
There are two ways to check if Availability service is functioning correctly:
Event Log4001 The Availability Service could not discover an Availability Service in the remote forest
4003 Public Folder Request Failed
4004 Unable to find a public folder server for the organizational unit
4005 Could not find information in Active Directory to allow cross-forest requests
4011 Cross-forestRequestFailed
Test-OutlookWebServices Cmdlet
Test-OutlookWebServices -id:[email protected] -TargetAddress: [email protected]
Microsoft Confidential48
Troubleshooting Availability Service, cont.
OWA vs Outlook 2007OWA runs against the Availability Service API(s) on the CAS server
OUTLOOK 2007 runs against the Availability Service Web Service and relies on the Autodiscover service to find the Availability URL
Mostly, free/busy problems in Outlook 2007 might be related with configuration of Autodiscover rather than Availability Service
Following commands can be used to get more information:Get-webservicesvirtualdirectory
Get-WebServicesVirtualDirectory -Identity EWS(default web site)
Get-WebServicesVirtualDirectory -Identity CAS01
Microsoft Confidential49
Test-OutlookWebServices
Test-OutlookWebService is a diagnostic task to verify AutoDiscover , Availability Service, RPC/HTTP and OAB distribution configuration for connectivity onlyTest-OutlookWebServices -Identity <Alias, Domain\User or SMTP address> -ClientAccessServer <FQDN or NetBIOS name> -TargetAddress<Alias, Domain\User or SMTP address>
Scope can be set for: For an Individual User: -Identity <Alias, Domain\User or SMTP address>
For a specific CAS Server: -ClientAccessServer <FQDN or NetBIOS name>
Free/Busy queries: -TargetAddress < Alias, Domain\User or SMTP address>
Returns information about SSL Certificate problems
Determines the validity of the returned service URLs
The request is made for one day of free busy data and the data is not returned in the task output.
Microsoft Confidential50
Test-OutlookWebServices – Basic Functionality
Step 1: Get a user context
Step 2: Determine the Autodiscover URL
Step 3: Submit an Autodiscover request
Step 4: Validate that services exist
Step 5: Return results of all tests to the console in the form of events.
Microsoft Confidential52
Troubleshooting Free/Busy using Outlook 2007 Logging
Outlook 2007 can be used to troubleshoot problems with the Autodiscover service
The Availability service log files are located in the \Documents and Settings\ <username> \Local Settings\Temp folder
Three Log types:OOF (Out of Office)
MS (Meetings Suggestions)
FB (Free/Busy)
Example:
20070305-110303994-fb.log
Microsoft Confidential54
Free/Busy Log Files
Generated each time a user is added to the meeting request from Scheduling Assistant tab or for each request sent from the client
Includes GetUserAvailabilityRequest XML message
There are only three blocks of interest that contain the detail information needed for diagnostics:• MessageText – Contains information about the failure.
• ExceptionCode– Contains the exception that caused the failure.
• ResponseCode– Contains the web response code for the failure.
Microsoft Confidential56
Troubleshooting Free/Busy specific failures
Mailbox Logon Failure : Check the status of the target user's mailbox to see if it is available
<MessageText>Mailbox logon failed., inner exception: Cannot open mailbox /o=Fourthcoffee/ou=Exchange Administrative Group (FYDIBOHF23SPDLT)/cn=Recipients/cn=mod4user11.</MessageText><ResponseCode>ErrorMailboxLogonFailed</ResponseCode>
Permissions Error : Engage the target mailbox owner to confirm calendar permissions
<MessageText>Caller does not have access to free busy data. </MessageText>
<ResponseCode>ErrorNoFreeBusyAccess</ResponseCode>
Microsoft Confidential57
Troubleshooting Free/Busy specific failures – cont.
Proxy Failures : The configuration need to enable cross-forest sharing of free/busy information is incomplete or miss-configured
<MessageText>The proxy request failed because the remote server returned an error…………
<ResponseCode>ErrorProxyRequestProcessingFailed</ResponseCode>
Legacy Free/Busy Failures : Public folder store on Exchange 2003 is not mounted or inaccessible
<MessageText>The remote server returned an error: (503) Server Unavailable..
<ResponseCode>ErrorPublicFolderRequestProcessingFailed</ResponseCode>
Microsoft Confidential58
Availability Service Error Codes
Description of Exchange 2007 Availability Service Error CodesRequestStreamTooBig = 5000
IdentityArrayEmpty = 5001
IdentityArrayTooBig = 5002
TimeIntervalTooBig = 5003
InvalidMergedFreeBusyInterval = 5004
ResultSetTooBig = 5006
InvalidClientSecurityContext = 5007
MailboxLogonFailed = 5008
MailRecipientNotFound = 5009
InvalidTimeInterval = 5010
PublicFolderServerNotFound = 5011
InvalidAccessLevel = 5012
And more…
Microsoft Confidential60
Agenda
Introduction
Overview
Locating CAS Configuration and Topology Data
Troubleshooting:
Autodiscover
Availability Service
Offline Address Book
Client Access Security
Outlook Web Access
Exchange ActiveSync
Microsoft Confidential61
Offline Address Book (OAB)
Used for offline GAL view and cached mode GAL lookups
New OAB uses HTTPs & BITS (Background Intelligent Transfer Service)
Legacy support available by using public folders distibution
Relies on:
OABGen
Exchange File Distribution
OAB Virtual Directory
Autodiscover
BITS Client does not support self-signed certificates, so by default OAB Distribution Points use HTTP
SSL can be enabled with a fully trusted certificate in IIS
Microsoft Confidential62
Offline Address Book - Process
Microsoft Confidential64
Troubleshooting Offline Address Book
Legacy clients
• Need to publish OAB to public folders
• Need OAB Public folder replication
Outlook 2007
• Make sure Autodiscover works and the URL’s are correct
• Check the OAB distribution on nearest CAS server
• Check IE proxy settings (KB939765)
Non client specific issues
• GAL deleted
• No permissions
Microsoft Confidential66
Agenda
Introduction
Overview
Locating CAS Configuration and Topology Data
Troubleshooting:
Autodiscover
Availability Service
Offline Address Book
Client Access Security
Outlook Web Access
Exchange ActiveSync
Microsoft Confidential68
SSL handshake
Microsoft Confidential69
SYNC (TCP_Port = 443)
SYNC + ACK
ACK
Outlook clicks on URL to
access Secure Webmail Server
(https://mail.msft.com)
The browser establishes a
TCP connection on the HTTPS TCP Port 443
CLIENT_HELLO
SERVER_HELLO
CERTIFICATE
SERVER_DONE
SSL Handshake on the new TCP connection
To continue with the authentication process, clientshould verify the server‟s certificate
1 Is today's date within the validity period?2 Is the issuing Certificate Authority (CA) a trusted CA?3 Does the issuing CA's public key validate the issuer's digital signature?
The server is authenticated. 5
Outlook Web Access Client Access Server
Does the domain name in the server's certificate match the domain name of the server itself?
4
Certificates and Subject Alternate Name
By default during CAS installation a new self signed certificate will be generated and assigned to the Default Web Site for encrypted HTTP communication
The authenticity of self signed certificate cannot be verified during Internet access
For Internet access to CAS Servers, it is recommended to use a 3rd party CA for authenticity to work
However the certificate which was assigned to company name may not match the CAS Web Address
Subject Alternate Names can be used as an alternative web address for an existing certificate to override that problem
Microsoft Confidential71
Certificate Request
Microsoft Confidential72
Self-signed Certificates & Outlook Anywhere
Microsoft Confidential74
Client Access Server
Outlook Provider
CertPrincipalname =
Null
By default, the
CertPrincipalName
parameter for
OutlookProvider is
not configured
Outlook Anywhere
OL uses the
ExternalHostname
parameter for
OutlookAnywhere to
populate the server name
listed after “MSSTD:”
By default, the
ExternalHostname parameter
will not match the default
“Issued To” value on the self-
signed certificate
If you use self-signed certificates, OL will not successfully connect using HTTPS by using default settings pushed by Autodiscover
Troubleshooting Self-signed Certificates
There are two methods to solve self-signed certificate problems with Outlook Anywhere:
Get a new certificate where the “Issued To” property matches the Certificate Principal Name
-OR-
Change the CertPrincipalName Value on OutlookProvider:
set-outlookprovider -identity EXPR -server 'owatest.mail.msft' –CertPrincipalName 'msstd:owatest.mail.msft'
This allows OL2007 to get complete the Autodiscover phase of Outlook Anywhere profile creation
Domain-joined clients do not display Invalid CA certificate warnings
Note: Self signed certs would generate warnings for end user and we recommend our customers to buy the required certificates before deploying CAS for the end-users.
Microsoft Confidential75
Certificate Issues for Combined CAS&HUB Role
Events 1037 and 2019 if third party certificateIs not enabled for SMTP Service on CAS/Hub Server –and-
Includes Netbios name in the Certificate Request
To fix this:If the domain name parameter includes Netbios or Server FQDN in thecertificate request, the certificate should be enabled for SMTP service.
Run Enable-ExchangeCertificate command to enable for SMTP
Alternatively, do not use Netbios or Server FQDN in the certificate request, use only Public FQDN
Microsoft Confidential76
Common Certificate Issues
Using the self signed certificate
• The certificate common name is the server NetBIOS name
• There is no automatic way to make the self signed certificate trusted
Using the old exchange 2003 certificate
• Does not have Autodiscover URL
Using new certificate without considering Autodiscover
ISA server is not publishing Autodiscover URL
Microsoft Confidential78
Troubleshooting CAS Security & Certificate Issues
Tools Components
Get-ExchangeCertificate Command used to view certificates from the local certificate store
EXTRA Tracing Enable the following components/tags:•Common\Certificate Validation•Networking Layer\Certificate •Transport\Certificate
Protocol Logging Tool to troubleshoot transport security problems
IIS Log Files Access Used to troubleshoot authentication errors while accessing web service or any other web page
Certlib Unofficial utility to dump themsExchServerInternalTLSCert value in AD into a readable format
Microsoft Confidential80
Outlook Anywhere– Authentication methods after SP1
Exchange 2007 RTM
Mandatory Parameter: ExternalAuthenticationMethod
Used to update OL2007 clients using Autodiscover service
Basic and NTLM Authentication methods were always reenabled on /rpc virtual directory regardless of this parameter
Exchange 2007 SP1
Ability to choose the authentication methods
New parameters:
ClientAuthenticationMethod
IISAuthenticationMethods
DefaultAuthenticationMethod
set-OutlookAnywhere -IISAuthenticationMethods <Basic or NTLM>
Microsoft Confidential82
Exercise 1: Introduction to SSL Certificates
Exercise 2: Subject Alternative Names (SAN)
Microsoft Confidential85
LAB 1 Troubleshooting Certificates
Exercise 1: Understanding the AutoDiscover Service
Exercise 2: Configuring the AutoDiscover Service for use by external Outlook clients
Microsoft Confidential86
LAB 2 Troubleshooting Autodiscover
Agenda
Introduction
Overview
Locating CAS Configuration and Topology Data
Troubleshooting:
Autodiscover
Availability Service
Offline Address Book
Client Access Security
Outlook Web Access
Exchange ActiveSync
Microsoft Confidential87
Outlook Web Access – Overview
Microsoft Confidential88
Exchange 2003 Mailbox
OWA 2003 rendering
Exchange 2007Mailbox
SharePoint &File Shares
Active Directory
HTTP toExchange 2003
OWA &WebDAV
HTTP toExchange 2007
OWA
Client Access Server
OWA
Auth
ISAPI
IIS
SSL
/owa
/exchange
/exchweb/public
OWA
2007
rendering
Spell-
Checkers
Active
Directory
Driver
Doc. HTML
transcoder
CAS
business
logic
Outlook
Web Access
Proxy
Authentication Methods
Authentication Method
Security LevelHow Passwords
Are SentClient Requirements
Basic authentication
Low (unless Secure Sockets Layer [SSL] is enabled)
Base 64-encoded clear textAll browsers support Basic authentication
Digest authentication
Medium HashedMicrosoft Internet Explorer® 5 or laterversions
Integrated Windows authentication
High
•Hashed when IntegratedWindows authentication isused•Kerberos ticket when Kerberos is used•Integrated Windows authentication includes the Kerberos and NTLM authentication methods
•Internet Explorer 2.0 or later versions forIntegrated Windows authentication;•Microsoft Windows 2000 Server or later versions with Internet Explorer 5 or later versions for Kerberos
Forms-basedauthentication
High Uses cookies to help secure a user's name and password
Internet Explorer
Microsoft Confidential90
Troubleshooting OWA – FBA Login
Microsoft Confidential91
OWA Client CAS Server
Anonymous GET /owa
Owaauth
.dll
Intercept
Redirect to owa/auth/logon.asp
Anonymous GET /owa/auth/logon.asp
Return FBA logon page
POST including username + password
Redirect to /owa + set Auth Cookie Auth Cookie
Authenticated Request GET /owaAuth Cookie
Owaauth
.dll
Set Cookie
Owaauth
.dll
Authenticated
Troubleshooting OWA: Tools
Exchange Management Shell (EMS)Get-OwaVirtualDirectory (Set/Remove/New)
Test-OwaConnectivity Cmdlet
Test-OwaConnectivity –ClientAccessServer:ServerName
Test-OwaConnectivity -URL:https://mail.domain.com/owa -MailboxCredential:(get-credential DomainName\AccountName)
Get-CASMailbox (Set)
Get-casmailbox UserName | fl owa*
Exchange Managment Console (EMC)
Microsoft Confidential93
Note: Looking for a specific command? Use Get-Help with correct wildcards
Example: get-help *OWA*
Troubleshooting OWA: Tools – cont.
Internet Information managerMSExchangeOWAAppPool must be started using Local System identity
Web Services Extensions must be enabled for: ASP.NET (), Microsoft Exchange Client Access Server (owaauth.dll)
CAS Only: Microsoft Exchange Server (exprox.dll)
CAS + MBX: Microsoft Exchange Server (davex.dll)
Check Mapped application for Legacy Virtual Directories (/Exchange, /Public and /EXCHWEB)
CAS Only: exprox.dll
CAS + MBX: davex.dll
Check Authentication settings (change using EMC or EMS)
Anonymous for owa/auth folder
Microsoft Confidential95
Troubleshooting OWA: Configuration
Check Configuration filesForms Registry file: Registry.xml
Web.Config
Registry KeysDisable LDAP Encryption (Troubleshooting ONLY)
Key: HKLM\SYSTEM\CurrentControlSet\Services\MSExchangeADAccess
DWORD: Disable LDAP Encryption
Value: 1 = LDAP Encryption disabled
Microsoft Confidential96
Troubleshooting OWA: Microsoft Fiddler HTTP debugger
Fiddler is a HTTP Debugging Proxy which logs all HTTP trafficInternet Explorer Plug-In
View HTTP(S) traffic (real-time)
HTTP(S) Statistics
Capturing web traffic logs
Session Inspector
RAW view of HTTP traffic
And many more
Download available at http://www.fiddlertool.com/fiddler/
IMPORTANT: Fiddler is a Client Side debugging Tool and should never be installed onto a production Exchange server. Serious problems have been reported with Active Sync when Fiddler is installed on an Exchange 200x server!
Microsoft Confidential98
Understanding Proxying and Redirection
Microsoft Confidential99
Internet
OWA EAS
CAS in
User’s mailboxAD Site
Find a CAS
In MailboxAD Site
Best CAS
has ExternalURLSet?
REDIRECT Client
using ExternalURL
PROXY request to
CAS using InternalURL
Execute request
on this CAS
OWA/EAS
not available
Note: If the mailbox is on E2K3, the
CAS will proxy directly to the back-end. Integrated Windows authentication for /Exchange and /Microsoft-Server-
ActiveSync virtual directories must be enabled
Note: CAS to CAS proxying is not
supported between virtual directories that use Basic authentication, the virtual directories must use Integrated
Windows authentication.
Yes
No
Yes
YesNo
No
Note: Redirection is only
supported for OWA
Understanding Proxying and Redirection - cont.
Microsoft Confidential101
CAS->MBX comm. Between
AD sites
RedirectionCAS->CAS
CAS-> proxying between AD sites
Comments/Consequences
OWA No Yes Yes
Must have a CAS server in each Exchange AD site to use OWA/EAS/Web Services
EAS NoUnnecessary: Autodiscover
Yes
Web Services used by 3rd party LOB applications
No No Yes
Availability Service used by Outlook 2007
NoUnnecessary: Autodiscover
Yes
Outlook Anywhere Yes, RPCUnnecessary: Autodiscover
Not applicable
WebDAV and OWA 2000/2003
Yes, HTTP No Not applicableProxying to legacy E2003 server
IMAP4/POP3 No No No
IMAP/POP clients must access a CAS in the mailbox AD Site directly
CAS Proxy Scenarios
Between Exchange 2007 Client Access ServersInternet facing CAS proxy requests to other CAS with no Internet presence
Known as CAS-CAS proxying
Between an Exchange 2007 Client Access Server and an Exchange Server 2003 Back-end server when:
OWA Clients connect to /Exchange virtual directory
EAS Clients connect to /Microsoft-Server-ActiveSync virtual directory
Microsoft Confidential103
Issues with Coexistence and DAVEX
When CAS and Mailbox roles are combined together, OWA clients are prompted for credentials two times or redirected to a different server to access an Exchange 2003 mailbox
For script mapping, /Exchange virtual directory will use Davex.dll instead of Exprox.dll
Davex.dll cannot act as a proxy for mailbox requests
Instead it redirects the requests to the Exchange 2003 Mailbox Server based on the internal (intranet) name of the server
External users will get DNS errors if internal name is not exposed to Internet
Microsoft Confidential104
Troubleshooting Proxying - Redirection
Verify Configuration using Exchange Management ShellInternalURL and ExternalURL values
RedirectToOptimalOWAServer=$True
Certificates SelfSigned/Public/Private vs
Set Diagnostic Logging for MSExchangeOWA\Proxy
set-EventLogLevel -Identity "ServerName\MSExchange OWA\Proxy" -Level "Expert"
Microsoft Confidential106
Proxying Performance and Scalability & Debugging
ASP.NET Proxying performance and scalability (KB821268)
Debug Registry Keys:
Allow Proxying without SSLRegistry Key: AllowProxyingWithoutSSL = 1
Allow Proxying only using Trusted Certificate
Registry Key: RequireTrustedCertForProxying = 1
Microsoft Confidential108
Exercise 1: CAS to CAS proxying
Exercise 2: Configuring OWA Redirection
Microsoft Confidential110
LAB 3 Troubleshooting Proxying
Web Ready Document Registry Keys
Set in the registry of the Client Access server under:
HKLM\SYSTEM\CurrentControlSet\Services\MSExchange OWA
Microsoft Confidential111
Value Name Value
Type
Value Default Value
RecycleByConversions DWORD # > 0 1000
ExcelRowsPerPage DWORD # > 0 200
MaxDocumentInputSize DWORD 1000000 > # > 0 5000 (in KB)
MaxDocumentOutputSize DWORD 1000000 > # > 0 5000 (in KB)
TempFolderLocation String Valid Path %SYSTEMROOT%\Temp
CacheDiskQuota DWORD # > 0 1000 (in MB)
ConversionTimeout DWORD # > 0 20 (seconds)
Troubleshooting WebReady Document Viewing
Microsoft Confidential113
Exchange Management Shell (EMS)Get-OwaVirtualDirectory (Set/Remove/New)
Get-CASMailbox (Set)
Exchange Managment Console (EMC)
ADSIEditCN=owa (Default Web Site),CN=HTTP,CN=Protocols, CN=<ServerName>, CN=Servers,CN=Exchange Administrative Group (FYDIBOHF23SPDLT),CN=Administrative Groups,CN=<Exchange Organization>,CN=Microsoft Exchange,CN=Services, CN=Configuration,DC=<Domain>,DC=<Domain>
Exercise 1: Examining the OWA login process (Forms based Authentication)
Exercise 2: WebReady Document viewing
Microsoft Confidential114
LAB 4 Troubleshooting OWA
Agenda
Introduction
Overview
Locating CAS Configuration and Topology Data
Troubleshooting:
Autodiscover
Availability Service
Offline Address Book
Client Access Security
Outlook Web Access
Exchange ActiveSync
Microsoft Confidential115
Exchange ActiveSync Overview
Synchronization protocol optimized to work with high-latency and low-bandwidth networks
Based on HTTP and XML
Enables mobile device users to access their e-mail, calendar, contacts and tasks
Enhanced in Exchange Server 2007:
Microsoft Confidential116
Support for HTML messages
Support for follow-up flags
Support for fast message retrieval
Meeting attendee
information
Enhanced Exchange Search
Windows SPS and LinkAccess
document accessPIN reset
Enhanced device security through
password policies
AutoDiscover for over the air provisioning
Support for OOF configuration
Support for tasks synchronization
Troubleshooting Exchange ActiveSync
Coexistence
Exchange 2003 Back end authentication (KB937031)
Read IIS logs
Use logparser
Certificates
Use device emulator
EXTRA Logging
Microsoft Confidential117
All Troubleshooting Tools
IIS Logs
LDAP tools (LDP, ldifde, adsiedit, etc …)
Event Logging
Performance Monitoring
Exchange Troubleshooting Assistant (Supervised Trace Logs)
Browser
Outlook logging
Microsoft Confidential118
Additional resources
Support WebCast: Introduction to AutoDiscover in Microsoft Exchange Server 2007http://support.microsoft.com/kb/935438
White Paper: Exchange 2007 Autodiscover Servicehttp://technet.microsoft.com/en-us/library/59adba4e-44e1-4aa2-b09d-06988cbeab2d.aspx
Autodiscover and Exchange 2007http://technet.microsoft.com/en-us/library/7c44814d-bb46-4fb8-9b6b-a082be35afdc.aspx
Managing the Autodiscover Servicehttp://technet.microsoft.com/en-us/library/aa995956.aspx
Exchange 2007 Autodiscover and certificates http://msexchangeteam.com/archive/2007/04/30/438249.aspx
More on Exchange 2007 and certificates - with real world scenariohttp://msexchangeteam.com/archive/2007/07/02/445698.aspx
Exchange 2007 Offline Address Book Web Distributionhttp://msexchangeteam.com/archive/2006/11/15/431502.aspx
Microsoft Confidential120
Troubleshooting Hub Transport Server (HT)
Agenda
Troubleshooting Mail flow Local Delivery & Mail Submission: Architecture
Local Delivery & Mail Submission: Troubleshooting
Remote Delivery: Architecture
Troubleshooting Mail Flow
Troubleshooting Mail QueuesMail Queues: Architecture
Mail Queues: Queues
Mail Queues: Troubleshooting
Troubleshooting Transport Certificates Certificates and Transport: Overview
Certificate Selection
Troubleshooting STARTTLS
Internal Transport
Troubleshooting Direct Trust
SP1 Changes
122
Local Delivery & Mail Submission: Architecture
Local DeliveryProcess by which messages are delivered from a Hub Transport server to mailbox on a local Mailbox server
Mail Submission
Process of picking up a message from a user’s mailbox and getting it into the Submission queue on a local Hub Transport server
Multiple components involvedExchange System Objects (XSO)
Microsoft Exchange Mail Submission Service
Store Driver
123
Local Delivery & Mail Submission: Architecture
Store DriverRuns on Hub Transport Server
Submits and retrieves mail to and from a local Mailbox server
Performs MAPI to MIME and MIME to MAPI conversion
Two primary components
Mail Submission RPC Server
Used to accept New Mail Notifications from the Mail Submission Service on Mailbox Server
XSO
Used to submit and retrieve mail to and from a Mailbox Server using MAPI.Net over RPC
Performs the MAPI to MIME and MIME to MAPI conversion
124
Local Delivery & Mail Submission: Architecture
Mail Submission ServiceRuns on Mailbox server
Notifies Hub Transport server of new messages for delivery
Two Primary components
Assistant Infrastructure
Processes and manages events that occur within the Information Store
Receives “Event” notifications about new messages for delivery
Triggers Mail Submission Service to generate new message notifications for Store Driver
Mail Submission RPC Client
Connects to Mail Submission RPC Server on Store Driver
Submits New Mail Notifications generated by Mail Submission Service
125
Local Delivery & Mail Submission: Local Delivery
Local delivery processOne Queue per Mailbox Server
Does not require Mailbox Store to run on local server
Does not use hidden SMTP Mailbox for conversion
No Store Driver on Mailbox Server
126
Mailbox Role
Hub Transport Role
EdgeTransport.exe
Store.exe
Outlook
Client
MAPI
JET
STORE STORESTORE
StoreDriver
XSOMailSubmission
RPC Server
RPC
Submission
Queue
Local
Delivery
Queue
CAT
MailSubmissionSvc.exe
Assistant
Infrastructure
(AI)
MailSubmission
RPC Client
MAPI.
Net
RPC
Dumpster
Message submission is completed Convert to
MAPI
Mailbox is in
local AD Site
Local Delivery & Mail Submission: Mail Submission
Mail Submission process
127
Mailbox Role
Hub Transport Role
EdgeTransport.exe
Store.exe
Outlook
Client
MAPI
JET
STORE STORESTORE
StoreDriver
XSOMailSubmission
RPC Server
RPC
Submission
Queue
Local
Delivery
Queue
CAT
MailSubmissionSvc.exe
Assistant
Infrastructure
(AI)
MailSubmission
RPC Client
MAPI.
Net
RPC
Dumpster
Locate Hub
Server
New Message Notification
Mailbox Server
DN, Sender„s MB
GUID, Message„s
EntryID, etc
Convert to MIME
Move message from Outbox to
Sent Items
Agenda
Troubleshooting Mail flow
Local Delivery & Mail Submission: Architecture
Local Delivery & Mail Submission: Troubleshooting
Remote Delivery: Architecture
Troubleshooting Mail Flow
Troubleshooting Mail QueuesMail Queues: Architecture
Mail Queues: Queues
Mail Queues: Troubleshooting
Troubleshooting Transport Certificates Certificates and Transport: Overview
Certificate Selection
Troubleshooting STARTTLS
Internal Transport
Troubleshooting Direct Trust
SP1 Changes
128
Local Delivery & Mail Submission: Troubleshooting
Run the Exchange Mail Flow Troubleshooter
Check Queues on the Hub server
Run Test-ServiceHealth on Mailbox and Hub server
Check event logs on Mailbox and Hub Server
Test basic RPC connectivity between Hub and Mailbox server
Enable connectivity logging
Verify AD site configuration
Verify DNS configuration
Check permissions for the Exchange Servers group on the Mailbox server object
129
Local Delivery & Mail Submission: Troubleshooting
Run the Exchange Mail Flow TroubleshooterAvailable in the EMC Toolbox
Analysis is symptom specific, so choice of correct symptom is critical to success
130
Inbound
NDR
Outbound
Queue
Submission
EdgeSync
Local Delivery & Mail Submission: Troubleshooting
Check Queues on the Hub serverUse Queue Viewer or the Get-Queue cmdlet to check the Status and LastError fields of the Mailbox Delivery queue
If Status is Retry then LastError should be an error message that can be used to help identify the problem
Run Test-ServiceHealth cmdlet on Mailbox and Hub serverEnsures all required services configured to start automatically have started
Returns an error for any service that is required by a configured role and is set to start automatically but is not currently running
131
Local Delivery & Mail Submission: Troubleshooting
Check event logs on Mailbox and Hub ServerCheck both Mailbox and Hub server for errors or warnings
If necessary increase diagnostics logging using the registry or Set-EventLogLevel
Mailbox server: MSExchangeMailSubmission\General
Hub Transport server: MSExchange Store Driver\General
The possible logging levels that you can set are:
0 (Lowest), 1 (Low), 3 (Medium), 5 (High), 7 (Expert)
Always return the logging level to the default setting after completing your troubleshooting activities
132
Local Delivery & Mail Submission: Troubleshooting
Test basic RPC connectivity between Hub and Mailbox serverUsing RPCPing
Two components
Rpings.exe (server-side RPC ping utility)
Rpingc.exe (client-side RPC ping utility)
Verify RPC connectivity in both directions
Using Test-MAPIConnectivity cmdlet
Logs onto the system mailbox or a specified mailbox using the -Identity parameter
Verifies that the MAPI server, Exchange store, and Directory Service Access (DSAccess) are working
133
Local Delivery & Mail Submission: Troubleshooting
Test-MailflowTests mail submission, transport and delivery
Tests services by verifying that each mailbox server can successfully send itself a message.
Remote functionality to test between remote Mailbox Servers
134
Local Delivery & Mail Submission: Troubleshooting
Enable connectivity loggingWhen the Hub server is having problems sending to the Mailbox server
Not enabled by default
get-transportserver|set-transportserver -ConnectivityLogEnabled:$TRUE
Provides summary information on outbound connections via SMTP or StoreDriver
Much less verbose than Protocol logs
Protocol logs more useful for SMTP conversation
Useful when wanting to see connectivity issues with the sending StoreDriver as protocol logs do not capture this information
Multiple events/connection, each using same connection ID for correlation
135
Local Delivery & Mail Submission: Troubleshooting
Verify AD site configurationEnsure AD sites and subnet configuration is correct
Run Nltest.exe /dsgetsite on Hub and Mailbox servers to verify they are in the same AD site
Beware: Changing IP address of Hub server may cause it to fall under another AD site and therefore leave Mailbox server in an AD site with no other Hub servers
136
Local Delivery & Mail Submission: Troubleshooting
Verify DNS configurationCorrect AD site determination is heavily dependant on correct DNS configuration
Verify Hub and Mailbox servers are configured with the correct internal DNS server
Ensure correct AD information such as domain controller SRV and A records as well as correct AD site information is stored in DNS
Use DCDIAG /TEST:DNS on domain controllers (use /X and /XSL to log to XML for easier reading)
Ensure correct A records are stored in DNS for Hub and Mailbox servers
Nslookup can be used to help verify the above
137
Local Delivery & Mail Submission: Troubleshooting
Check permissions for the Exchange Servers group on the Mailbox server object
Using ADSIEdit.msc verify that the Exchange Servers group has an explicit allow set on the Mailbox server object for
Store Constrained Delegation
Store Read and Write Access
Store Read only Access
Store Transport Access
Ensure there are no explicit denies set as this will override the allow
Hub server requires these permissions in order to retrieve messages from and submit messages to mailboxes
138
Agenda
Troubleshooting Mail flow
Local Delivery & Mail Submission: Architecture
Local Delivery & Mail Submission: Troubleshooting
Remote Delivery: Architecture
Troubleshooting Mail Flow
Troubleshooting Mail QueuesMail Queues: Architecture
Mail Queues: Queues
Mail Queues: Troubleshooting
Troubleshooting Transport Certificates Certificates and Transport: Overview
Certificate Selection
Troubleshooting STARTTLS
Internal Transport
Troubleshooting Direct Trust
SP1 Changes
139
Remote Delivery: Architecture
Remote Delivery refers to messages delivered using SMTP toAnother Hub server in the organization
An Edge server
Another remote SMTP mail system
Can also be to a foreign mail system using the Drop directory
140
Remote Delivery: Architecture
DNSExchange 2007 uses an enhanced DNS client to resolve the next hop selection to a list of target server names
Standard DNS client used to resolve list of server names to IP addresses
Enhanced DNS also provides load-balancing for Hub servers by using round robin
SMTPUsed for communication when messages are relayed between SMTP servers
141
Remote Delivery: Architecture
Routing TablesHolds information that the routing component uses to make routing decisions
Composed of a map of topology components and their relationship to one another
Linked connectors map
Server map
Legacy server map
MDB map
Active Directory site map
Routing groups map
Send connectors map
142
Remote Delivery: Routing Architecture
Routing TablesBuilt every time that a transport server is started
Recalculated when configuration changes are received
Configuration changes can be detected in the following ways
Active Directory change notifications
Configuration reloading caused by service control commands
Periodic reload to track changes that are not supported by Active Directory notifications
Information in the routing tables is logged to routing logs
C:\Program Files\Microsoft\Exchange Server\TransportRoles\Logs\_ Routing
New log generated every time routing tables are calculated
If the Hub server is unable to contact AD routing will use currently cached routing data
144
Remote Delivery: Architecture
Viewing the Routing Table LogRouting Log Viewer used to view the Routing Table Log
New tool with Microsoft Exchange Server 2007 SP1
145
Can be used to
Find the Lowest Cost Path to a Site
Find the Preferred Connector for a Given Address
Can open a second routing log and determine changes that have occurred within the routing topology between two time periods
Remote Delivery: Demo
In this demo your instructor will show youHow to use the Routing Log Viewer tool to
Find the lowest cost path to a site
Find the preferred connector for a given address space
Compare two routing logs to identify changes
147
Remote Delivery: Architecture
Determining Site MembershipNetLogon service determines site membership for the computer upon startup
Uses DNS queries to compare the local IP address to defined subnets
Exchange AD Topology service retrieves the site membership value from the NetLogon service
msExchServerSite attribute
Value of this attribute is the DN of the AD site of an Exchange server
Reduces overhead associated with DNS queries
Also used to associate a non-domain computer, such as a subscribed Edge server to an AD site
Populated and kept up to date by the Exchange AD Topology service
148
Remote Delivery: Architecture
Detecting Site Membership ChangesMay occur due to IP address change or AD subnet association change
Exchange 2007 must update its configuration data so that the change is considered when making routing decisions
The NetLogon service polls frequently for changes in AD site membership
The Exchange AD Topology service queries NetLogon regularly to determine the AD site membership of the local Exchange server and updates the MsExchServerSite attribute if necessary
The Exchange servers in the organization update the routing tables with the new AD site membership attribute value
150
Remote Delivery: Architecture
Controlling IP Site Link CostsBy default, Exchange Server 2007 uses the cost that is assigned to the AD IP site links
You can use the Set-AdSiteLink cmdlet to assign an Exchange-specific cost to an IP site link connector
Exchange-specific cost is a separate attribute and overrides the AD assigned cost for the purpose of determining the Exchange routing path
Allows the Exchange administrator to override existing links however it does not allow the ability to create links where none exist
Useful when the AD IP site link costs do not result in an optimal Exchange message routing topology
Using it for permanently overriding costs simply adds complexity
Better to work with AD and network administrators
152
Remote Delivery: Architecture
Message size limit for IP site linksNew to Exchange 2007 SP1
By default Exchange does not impose a maximum message size limit on messages that are relayed between Hub servers in different AD sites
Useful when low-bandwidth connections to remote sites exist
Use the Set-AdSiteLink cmdlet to configure
Set-AdSiteLink -Identity <IP Site link name> -MaxMessageSize 10MB
153
Remote Delivery: Troubleshooting
Back PressureBack pressure is a system resource monitoring feature of the Exchange Transport service
If utilization of a system resource exceeds the specified limit the server stops accepting new connections and messages
Prevents the system resources from being completely overwhelmed and enables the server to deliver the existing messages
When utilization returns to a normal level the server accepts new connections and messages
For each monitored system resource the following three levels of resource utilization are applied:
Normal
Medium
High
154
Agenda
Troubleshooting Mail flow
Local Delivery & Mail Submission: Architecture
Local Delivery & Mail Submission: Troubleshooting
Remote Delivery: Architecture
Troubleshooting Mail Flow
Troubleshooting Mail QueuesMail Queues: Architecture
Mail Queues: Queues
Mail Queues: Troubleshooting
Troubleshooting Transport Certificates Certificates and Transport: Overview
Certificate Selection
Troubleshooting STARTTLS
Internal Transport
Troubleshooting Direct Trust
SP1 Changes
155
Troubleshooting Mail Flow
Back Pressure (Cont...)System resources monitored
Free space on the drive that stores the message queue database
Free space on the drive that stores the message queue database transaction logs
The number of uncommitted message queue database transactions that exist in memory
The memory that is used by the EdgeTransport.exe process
The memory that is used by all processes
156
Troubleshooting Mail Flow
Understanding Non-Delivery ReportsNDRs redesigned in 2007 to make them easier to read and understand
Separated into user information and diagnostic information for admins
157
Troubleshooting Mail Flow
Understanding Non-Delivery ReportsThe following fields are present in most NDRs
Generating server
Rejected recipient
Remote server
Enhanced status code
SMTP response
Original message headers
Enhanced status codes
Success is indicated by a 2.x.x status code
Persistent Transient Failure is indicated by a 4.x.x status code
Permanent Failure is indicated by a 5.x.x status code
158
Troubleshooting Mail Flow
Message Tracking LogsDetailed log of all message activity as messages are transferred
161
Enabled by default
EventID describes tracking event action
Source describes component involved
SMTP, STOREDRIVER, ROUTING, AGENT
Use Set-TransportServer or Set-MailboxServer cmdlets to configure
Troubleshooting Mail Flow
Message tracking log event typesBADMAIL
DELIVER
DEFER
DSN
EXPAND
FAIL
POISONMESSAGE
162
RECEIVE
REDIRECT
RESOLVE
SEND
SUBMIT
TRANSFER
Troubleshooting Mail Flow
Filtering message tracking logsSearch depends on the Microsoft Exchange Transport Log Search service
Get-MessageTrackingLog cmdlet or Message Tracking tool in EMC to filter and display results
Get-MessageTrackingLog -sender [email protected] -start “07/01/2007 09:00 AM” -end “07/01/2007 09:30 AM”
In SP1 use GetMessageTrackingLogE2EwithTime.ps1 to search for specific entries in all message tracking logs on all Hub and Mailbox servers
163
Troubleshooting Mail Flow
Protocol LogsSMTP protocol conversation without data
Useful for diagnosing SMTP mail flow problems
Use Set-ReceiveConnector or Set-SendConnector cmdlets or the EMC to enable
164
Troubleshooting Mail Flow
Troubleshooting mail flow issues between Hub serversUse EXMFA with symptom matching the case
Determine where message delivery failed or where a non-delivery report (NDR) is being generated by
Using the Mail Flow Troubleshooter tool
Using the Queue Viewer on a Hub server to determine where message delivery failed
Checking the NDR to verify which server and component are generating the NDR
Use message tracking in order to determine the path messages are taking and at what point delivery is failing
Check the delivery status notification (DSN) error codes contained in the NDR and search the Microsoft support site for the error code
Run the Test-ServiceHealth cmdlet on the Hub servers where message delivery failed or on the transport server that generated the NDR
165
Troubleshooting Mail Flow
Troubleshooting mail flow issues between Hub servers (Cont...)Check the application event log on the Hub Transport servers that are involved in the delivery of the message
Increase diagnostic logging levels on the Exchange processes that are generating errors if necessary (SMTP Send and Receive, RemoteDelivery, Routing etc)
Verify that back pressure is not occurring
Event ID 15001 and 15002 for RTM and 15004 and 15005 for SP1
Check Physical connectivity
From the server where message delivery is failing ping the next hop servers by IP and FQDN and ensure a reply is received
At the same time ensure that the correct IP is resolve
Run netstat –anb on the receiving Hub server and verify that MSExchangeTransport.exe is listening on port 25
166
Troubleshooting Mail Flow
Troubleshooting mail flow issues between Hub servers (Cont...)From the server where message delivery is failing verify you can connect to SMTP port 25 of the next hop servers by using telnet
Verify the necessary connectors are enabled and configured appropriately on the Hub servers involved
Default receive connector configuration, Remote IP range, message size restrictions, authentication methods and types
Verify AD site configuration
Ensure AD sites and subnet configuration is correct
Run Nltest.exe /dsgetsite on next hop Hub servers to verify they fall in the correct AD site
Check msExchServerSite attribute using ADSIEdit.msc for next hop Hub server objects to ensure they are stamped with the correct AD site
167
Troubleshooting Mail Flow
Troubleshooting mail flow issues between Hub servers (Cont...)Verify DNS configuration
Verify Hub servers are configured with the correct internal DNS server
Ensure correct information such as A records for next hop servers are stored in DNS
Ensure correct AD information such as domain controller SRV and A records as well as correct AD site information is stored in DNS
Use DCDiag /TEST:DNS on domain controllers
Ensure the correct A records are stored in DNS for the Hub servers
Nslookup can be used to verify the above
Enable protocol logging for the Send connector on the sending Hub server and Receive connector on the receiving Hub server
168
Troubleshooting Mail Flow
Troubleshooting mail flow issues between Hub servers (Cont...)Look for possible certificate issues
If custom certificates are installed ensure they are enabled for the SMTP service using Get-ExchangeCertificate
Use network monitor to capture network traffic between the sending and receiving server
Test-Mailflow
Tests mail submission, transport and delivery
Tests services by verifying that each mailbox server can successfully send itself a message.
Remote functionality to test between remote Mailbox Servers
169
Troubleshooting Mail Flow
Troubleshooting outbound internet mail flowFollow the same steps as when troubleshooting mail flow issues between hub servers
Ensure that an appropriate Internet send connector has been configured with a * address space
When the send connector is configured to use DNS
From the sending server use Nslookup to determine external email domains MX records
Set type=MX
From the sending server telnet to port 25 of the identified remote host and send a test message
When the send connector is configured to use a Smart Host
From the sending server telnet to port 25 of the Smart Host and send a test message
170
Troubleshooting Mail Flow
Troubleshooting inbound internet mail flowEnsure that an authoritative accepted domain has been configured for the email domain and has public MX records registered
Ensure that an email address policy has been enabled for the accepted domain and users have the email address applied
Run netstat –anb on the Hub or Edge servers responsible for receiving inbound internet mail and verify that MSExchangeTransport.exe is listening on port 25
Verify the public MX records registered for the receiving email domain using nslookup
Set type=MX
May need to point nslookup to an external DNS server
“server <External DNS Server IP>”
Using a machine on the internet telnet to port 25 of the server identified in the MX record and send a test message
171
Troubleshooting Mail Flow
Troubleshooting inbound internet mail flow (Cont...)Ensure that any firewalls are configured to forward inbound internet messages onto the Exchange servers responsible for inbound internet mail
Verify the necessary receive connectors are enabled and configured appropriately on the Hub or Edge servers involved
Default receive connector configuration, Remote IP range, message size restrictions, authentication methods and types (specifically anonymous connections)
For Hub servers default Receive connector will need to be modified to allow anonymous connections
172
173
LAB 5 Mailflow
Agenda
Troubleshooting Mail flow
Local Delivery & Mail Submission: Architecture
Local Delivery & Mail Submission: Troubleshooting
Remote Delivery: Architecture
Troubleshooting Mail Flow
Troubleshooting Mail Queues
Mail Queues: Architecture
Mail Queues: Queues
Mail Queues: Troubleshooting
Troubleshooting Transport Certificates Certificates and Transport: Overview
Certificate Selection
Troubleshooting STARTTLS
Internal Transport
Troubleshooting Direct Trust
SP1 Changes
174
Mail Queues: Architecture
Queues are temporary holding locations for messages that are waiting to enter the next stage of processing
Each queue represents a logical set of messages that a transport server processes in a specific order
Exist only on Hub Transport or Edge Transport server roles
Exchange 2007 all messages stored in the same location
All queues are stored in a single ESE database
175
Mail Queues: Architecture
ESE Database Queue filesESE Database files centrally located on the server
By default located at ...\Exchange Server\TransportRoles\data\Queue
Circular logging is used
Configuration options stored in EdgeTransport.exe.config
File Description
Mail.que ESE Database file that stores all the queued messages
Tmp.edb Temp database file used to verify the queue database schema
on startup
Trn*.log Transaction logs that record all changes to the queue
database
Trntmp.log Temporary transaction log created in advance
Trn.chk Tracks the log entries that have been committed to the
database
Trnres00001.jrs & Trnres00002.jrs
Transaction reserve log files. Used when the hard disk drive
that contains the transaction log runs out of space to stop the queue database cleanly
176
Agenda
Troubleshooting Mail flow
Local Delivery & Mail Submission: Architecture
Local Delivery & Mail Submission: Troubleshooting
Remote Delivery: Architecture
Troubleshooting Mail Flow
Troubleshooting Mail Queues
Mail Queues: Architecture
Mail Queues: Queues
Mail Queues: Troubleshooting
Troubleshooting Transport Certificates Certificates and Transport: Overview
Certificate Selection
Troubleshooting STARTTLS
Internal Transport
Troubleshooting Direct Trust
SP1 Changes
177
Mail Queues: Queues
Several queues in Exchange Server 2007Submission
Remote Delivery
Mailbox Delivery
Poison Message
Unreachable
178
Mail Queues: Queues
Submission QueueUsed by the categorizer to gather all messages that have to be resolved, routed, and processed by Transport agents
Only one Submission queue per transport server
Messages remain here until categorization is complete
Messages enter Submission Queue from various sources
179
Mail Queues: Queues
Messages may queue up in the Submission Queue due toIssues with AD such as slow DCs, AD permissions, unable to communicate with a DC
Problems with Transport Routing Agents
Performance issues on the server itself such as a disk or processor bottleneck
Large nested or query-based distribution groups
180
Mail Queues: Queues
Remote Delivery QueuesHandles the queuing of a message to a specific SMTP based target destination
Remote SMTP Host
Drop Directory
Queue for every remote domain
Created and deleted dynamically
Each remote delivery queue contains messages being routed to the same delivery destination
Edge Servers always places messages into Remote Delivery Queues unless they are poison or unreachable
181
Mail Queues: Queues
Messages may queue up in the Remote Delivery queue due toConnectivity issues such as network links being down or ports being blocked
DNS name resolution issues
Mis-configurations of receive connectors such as incorrect Authentication settings or Remote IP Ranges
Certificate issues
Performance issues on either the sending or receiving server
182
Mail Queues: Queues
Local Delivery QueueHolds messages being delivered to a Mailbox server in the local AD site using encrypted Exchange RPC
Only available on Hub Servers
Processed by Store Driver
More than one mailbox delivery queue can exist
183
Mail Queues: Queues
Messages may queue up in the Local Delivery queue due toConnectivity issues to the Mailbox server such as network links being down or ports being blocked
Services or stores being offline on the Mailbox server
DNS name resolution issues
Performance issues on the local transport or receiving Mailbox server
Unable to log onto mailbox due to incorrect permissions for the Mailbox server
184
Mail Queues: Queues
Poison Message QueueUsed to isolate messages that are detected to be potentially harmful
Contains messages that caused the Transport Worker Process to crash
Exchange 2003 messages could repeatedly crash SMTP Service
Required manual extraction
Exchange 2007 can detect if a message crashes Transport
Message is removed from processing and placed in the Poison Message Queue
Maintains “Poison Message Count” on each message
If count exceeds threshold, moved to Poison Message Queue
Threshold controlled by PoisonThreshold value on TransportServer settings
Default 2
Messages can be deleted by admin, exported for debug, resubmitted or expire
185
Mail Queues: Queues
Unreachable QueueContains messages that cannot be routed to their destinations
Messages that have no route due to configuration errors
Problems with connectors
Missing attributes on mail enabled objects (HomeMDB)
Messages without routes don’t NDR immediately they are placed in the Unreachable queue
Each transport server can have only one Unreachable queue
Messages are automatically resubmitted for categorization if routing topology changes are detected
186
Mail Queues: Queues
Scenarios where messages are placed into the Unreachable queue
The recipient is a valid Active Directory recipient object. However a routing path cannot be calculated for that recipient
The recipient is an external SMTP address and a matching connector cannot be found for the address space
The recipient is a distribution group and the expansion server for the distribution group is invalid or does not have the Hub Transport server role installed
The recipient is an SMTP address recipient of a message that was received on a Receive connector that is linked to a Send connector that is ignored by the routing component of the categorizer because it is disabled or misconfigured in some way
187
Mail Queues: Queues
Scenarios where messages are NOT placed into the Unreachable queue and an NDR is generated instead
The routing path cannot be calculated for a recipient because constraints, such as message size restrictions, prevent delivery of the message using the single, deterministic route calculated by the categorizer
The recipient is a non-SMTP address and a matching connector cannot be found. Or the matching connector is disabled or misconfigured
The recipient is a non-SMTP address recipient that was received on a Receive connector that is linked to a Send connector that is ignored by the routing component of the categorizer because the Send connector is disabled or misconfigured
188
Mail Queues: Queues
Queue message processing in RTMAll queues except for Submission queue process messages using First In/First Out (FIFO)
Submission Queue uses “Round Robin”
For every X number of higher priority messages processed process a lower priority message
Keeps higher priority messages from stopping delivery of low priority
Ratio cannot be modified
189
Mail Queues: Queues
Priority Queuing in Exchange 2007 SP1Priority queuing affects the transmission of messages from a delivery queue to the destination messaging server
When enabled higher priority messages are transmitted before lower priority messages to their destinations
Helps admins define specific SLA requirements for message delivery times
Enabled or disabled in EdgeTransport.exe.config
PriorityQueuingEnable parameter
When enabled the priority message queue limits (such as expiration of messages) in EdgeTransport.exe.config override the message queue limits set by Set-TransportServer
190
Mail Queues: Queues
QUEUING
OUTBOUND
INBOUND
SUBMISSION
CATEGORIZER
POISON
UNREACHABLE
REMOTE
MAPI
SMTPOut
SMTPIn
RESUBMIT
STOREDRIVER
PICKUP/REPLAY
DIRECTORY
AGENTS
191
Agenda
Troubleshooting Mail flow
Local Delivery & Mail Submission: Architecture
Local Delivery & Mail Submission: Troubleshooting
Remote Delivery: Architecture
Troubleshooting Mail Flow
Troubleshooting Mail Queues
Mail Queues: Architecture
Mail Queues: Queues
Mail Queues: Troubleshooting
Troubleshooting Transport Certificates Certificates and Transport: Overview
Certificate Selection
Troubleshooting STARTTLS
Internal Transport
Troubleshooting Direct Trust
SP1 Changes
192
Mail Queues: Troubleshooting
Troubleshooting mail queues involves looking at queue and message properties to help isolate the issue, specifically
Status
Either Active, Ready, Retry or Suspended
Next Hop Domain
Available for queues only
Shows the destination of the queue such as a Smart Host or another SMTP or mailbox server
Last Error
Lists the reason for message delivery failure
If a queue is or messages are in a retry state look at the Last Error value to help isolate the reason for failure
Use the Next Hop Domain value to help determine where the message is being delivered
193
194
LAB 6 Disaster Recovery
Agenda
Troubleshooting Mail flow
Local Delivery & Mail Submission: Architecture
Local Delivery & Mail Submission: Troubleshooting
Remote Delivery: Architecture
Troubleshooting Mail Flow
Troubleshooting Mail Queues
Mail Queues: Architecture
Mail Queues: Queues
Mail Queues: Troubleshooting
Troubleshooting Transport Certificates Certificates and Transport: Overview
Certificate Selection
Troubleshooting STARTTLS
Internal Transport
Troubleshooting Direct Trust
SP1 Changes
195
Certificate and Transport: Overview
SMTP STARTTLSMutual TLS
Opportunistic TLS
Force Required TLS
By Connector
Domain Security
POP
IMAP
Edge Synchronization
Certificate and Transport: Overview (cont.)
Direct Trust CertificateRetrieved from AD
Must be available also in local Certificate Store
SMTP X-AnonymousTLSHub to Hub
Hub to Edge/Vice Versa
Direct Trust Authentication
Microsoft Confidential198
Agenda
Troubleshooting Mail flow
Local Delivery & Mail Submission: Architecture
Local Delivery & Mail Submission: Troubleshooting
Remote Delivery: Architecture
Troubleshooting Mail Flow
Troubleshooting Mail Queues
Mail Queues: Architecture
Mail Queues: Queues
Mail Queues: Troubleshooting
Troubleshooting Transport Certificates
Certificates and Transport: Overview
Certificate Selection
Troubleshooting STARTTLS
Internal Transport
Troubleshooting Direct Trust
SP1 Changes
199
Certificate Selection - STARTTLS
Process that components go through to determine which certificate should be used for an incoming connection
FQDNAlways based upon the connector FQDN
Get-<SEND | RECEIVE>Connector | FL FQDN
Certificate Selection - STARTTLS (cont.)
List of valid Certificates are found based on the connector FQDN
Has either no extended key usage extension or has the extended key usage extension containing the Server Authentication Object IdentifierHas either no key usage extension or it has the key usage extension with digital signature bit assertedIt has a RSA public key > 1024 bits in sizeHas a valid certificate chain up to a trusted root (or self signed)Revocation checking passes on the certificate chainThe private key is present and accessible (Network Service)The private key is not stored in a removable deviceThe private key is not UI protected
Certificate Selection - STARTTLS
From the remaining list, pick the best certificate (in order of preference)
Trusted CA Issued Certificate preferred over Self-Signed
Newest installed certificate over oldest
An older 3rd Party CA issued certificate would be used over a newer self-signed certificate
Certificate Selection Process – Inbound STARTTLS
Microsoft Confidential203
POP3 and IMAP4
Selection process is similar to SMTP STARTTLS
Three Exceptions:
Instead of FQDN they use X509CertificateNameGet-POPSettings
Get-IMAPSettings
Newest valid certificate wins – changed in SP1
Does not support Wildcards such as *.fourthcoffee.com –changed in SP1
Agenda
Troubleshooting Mail flow
Local Delivery & Mail Submission: Architecture
Local Delivery & Mail Submission: Troubleshooting
Remote Delivery: Architecture
Troubleshooting Mail Flow
Troubleshooting Mail Queues
Mail Queues: Architecture
Mail Queues: Queues
Mail Queues: Troubleshooting
Troubleshooting Transport Certificates
Certificates and Transport: Overview
Certificate Selection
Troubleshooting STARTTLS
Internal Transport
Troubleshooting Direct Trust
SP1 Changes
205
Troubleshooting - STARTTLS
Tools UsedExchange Management Shell
TELNET.EXE (is STARTTLS advertised?)
TCP.EXE (What certificate is being served?)
CertUtil to verify certificates
Application Logs
Troubleshooting - STARTTLS
What is the FQDN of the Receive ConnectorGet-ExchangeCertificate –DomainName <FQDN>Get-ExchangeCertificate –DomainName <FQDN> | FL *Is there more than one certificate?
Are any certificates issued from a trusted CA?Are the certificates valid?What is the newest certificate?Can we access the CRL Distribution Point? Proxy?
Troubleshooting
[PS] C:\>Get-ExchangeCertificate 91BFBDD870D8928018E22B736922411645218B85 | fl *
CertificateDomains : {clt-e2k7.fourthcoffee.com, clt-e2k7}
CertificateRequest :
IisServices : {IIS://clt-e2k7/W3SVC/1}
IsSelfSigned : False
RootCAType : Enterprise
Services : IIS, SMTP
Status : Valid
PrivateKeyExportable : True
Archived : False
FriendlyName : Microsoft Exchange
IssuerName : System.Security.Cryptography.X509Certificates.X500Distin
guishedName
NotAfter : 9/17/2009 10:11:42 AM
NotBefore : 9/18/2007 10:11:42 AM
HasPrivateKey : True
SerialNumber : 610FCD9F000000000011
SubjectName : System.Security.Cryptography.X509Certificates.X500Distin
guishedName
Thumbprint : 91BFBDD870D8928018E22B736922411645218B85
Version : 3
Handle : 133224224
Issuer : CN=LON-E2K7, DC=fourthcoffee, DC=com
Subject : CN=clt-e2k7.fourthcoffee.com
Troubleshooting
[PS] C:\Get-ExchangeCertificate 6CC3257C2236DFC88BA40CD9A374C9E53CC18E2B | fl *
CertificateDomains : {clt-e2k7, clt-e2k7.fourthcoffee.com}CertificateRequest :
IisServices : {}
IsSelfSigned : True
RootCAType : Registry
Services : SMTP
Status : ValidPrivateKeyExportable : False
Archived : False
FriendlyName : Microsoft Exchange
IssuerName : System.Security.Cryptography.X509Certificates.X500Distin
guishedName
NotAfter : 9/19/2008 9:44:29 AM
NotBefore : 9/19/2007 9:44:29 AM
HasPrivateKey : TrueSerialNumber : 9816558F2C99EF924E7A5AA1730498DA
SubjectName : System.Security.Cryptography.X509Certificates.X500Distin
guishedName
Thumbprint : 6CC3257C2236DFC88BA40CD9A374C9E53CC18E2B
Version : 3
Handle : 68983992
Issuer : CN=clt-e2k7
Subject : CN=clt-e2k7
Troubleshooting
TCP.EXE
Troubleshooting
[PS] C:\>certutil -verify certnew.cerIssuer:
CN=LON-E2K7DC=fourthcoffeeDC=com
Subject:CN=clt-e2k7
Cert Serial Number: 610cbc3c000000000010
dwFlags = CA_VERIFY_FLAGS_DUMP_CHAIN (0x40000000)
ChainFlags = CERT_CHAIN_REVOCATION_CHECK_CHAIN_EXCLUDE_ROOT (0x40000000)HCCE_LOCAL_MACHINE
CERT_CHAIN_POLICY_BASE-------- CERT_CHAIN_CONTEXT --------ChainContext.dwInfoStatus = CERT_TRUST_HAS_PREFERRED_ISSUER (0x100)
ChainContext.dwErrorStatus = CERT_TRUST_REVOCATION_STATUS_UNKNOWN (0x40)ChainContext.dwErrorStatus = CERT_TRUST_IS_OFFLINE_REVOCATION (0x1000000)ChainContext.dwRevocationFreshnessTime: 4 Days, 22 Hours, 34 Minutes, 49 Seconds
SimpleChain.dwInfoStatus = CERT_TRUST_HAS_PREFERRED_ISSUER (0x100)
SimpleChain.dwErrorStatus = CERT_TRUST_REVOCATION_STATUS_UNKNOWN (0x40)SimpleChain.dwErrorStatus = CERT_TRUST_IS_OFFLINE_REVOCATION (0x1000000)SimpleChain.dwRevocationFreshnessTime: 4 Days, 22 Hours, 34 Minutes, 49 Seconds
CertContext[0][0]: dwInfoStatus=102 dwErrorStatus=1000040Issuer: CN=LON-E2K7, DC=fourthcoffee, DC=comSubject: CN=clt-e2k7Serial: 610cbc3c000000000010Template: WebServer06 99 41 18 54 db 2d 8b 2c ae 0a 5d d7 b5 27 54 42 d8 20 0b
Element.dwInfoStatus = CERT_TRUST_HAS_KEY_MATCH_ISSUER (0x2)Element.dwInfoStatus = CERT_TRUST_HAS_PREFERRED_ISSUER (0x100)
Element.dwErrorStatus = CERT_TRUST_REVOCATION_STATUS_UNKNOWN (0x40)
Certutil Sample :
Troubleshooting
Microsoft Confidential212
Element.dwErrorStatus = CERT_TRUST_IS_OFFLINE_REVOCATION (0x1000000)CRL 15:
Issuer: CN=LON-E2K7, DC=fourthcoffee, DC=com
93 a6 c0 1b ad cf 8f 9a 91 3b 6e b5 7e bc 93 ed 53 89 89 5c
Delta CRL 16:
Issuer: CN=LON-E2K7, DC=fourthcoffee, DC=com
35 9e 39 96 9d e8 08 ce 3c 16 a5 99 d5 aa 28 89 d1 54 db 3e
Application[0] = 1.3.6.1.5.5.7.3.1 Server Authentication
CertContext[0][1]: dwInfoStatus=10c dwErrorStatus=0
Issuer: CN=LON-E2K7, DC=fourthcoffee, DC=com
Subject: CN=LON-E2K7, DC=fourthcoffee, DC=com
Serial: 115643e01d0eab874e228cc4545d7e6c
d6 80 20 2f 11 ad f2 39 53 b5 92 df c1 5a 26 28 c4 5c e5 90
Element.dwInfoStatus = CERT_TRUST_HAS_NAME_MATCH_ISSUER (0x4)
Element.dwInfoStatus = CERT_TRUST_IS_SELF_SIGNED (0x8)
Element.dwInfoStatus = CERT_TRUST_HAS_PREFERRED_ISSUER (0x100)
Exclude leaf cert:
ef af 86 b5 a7 9e 25 51 44 18 98 b6 69 f9 df 62 c5 65 31 66
Full chain:
a8 51 bc 17 d8 b4 94 5a 6a 3d b9 01 89 bc c6 63 37 8e 0b ea
Issuer: CN=LON-E2K7, DC=fourthcoffee, DC=com
Subject: CN=clt-e2k7
Serial: 610cbc3c000000000010
Template: WebServer
06 99 41 18 54 db 2d 8b 2c ae 0a 5d d7 b5 27 54 42 d8 20 0b
The revocation function was unable to check revocation because the revocation se
rver was offline. 0x80092013 (-2146885613)
Revocation check skipped -- server offline
ERROR: Verifying leaf certificate revocation status returned The revocation func
tion was unable to check revocation because the revocation server was offline. 0
x80092013 (-2146885613)
CertUtil: The revocation function was unable to check revocation because the rev
ocation server was offline.CertUtil: -verify command completed successfully.
Certutil Sample cont: :
Agenda
Troubleshooting Mail flow
Local Delivery & Mail Submission: Architecture
Local Delivery & Mail Submission: Troubleshooting
Remote Delivery: Architecture
Troubleshooting Mail Flow
Troubleshooting Mail Queues
Mail Queues: Architecture
Mail Queues: Queues
Mail Queues: Troubleshooting
Troubleshooting Transport Certificates
Certificates and Transport: Overview
Certificate Selection
Troubleshooting STARTTLS
Internal Transport
Troubleshooting Direct Trust
SP1 Changes
213
Internal Transport - Certificate
The Direct Trust Certificate:Whenever X-AnonymousTLS is used
Hub to HubHub to EdgeEdge to Hub
Used to establish “Direct Trust” authentication after X-AnonymousTLS negotiation between a Hub and Edge server.Used to establish secure LDAP connections from Hub for Edge SynchronizationUsed to encrypt and decrypt EdgeSynchronization credentials which are stored in the directory.
Direct Trust Certificates = Default Certificate
Internal Transport - Certificate selection
The Internal Transport Certificate selection process simply loads from msExchServerInternalTLSCert property on the Exchange Server object
Must also be found in the local Computer Store
Expired Direct Trust Certificates do not affect mail flow
Warning:
Error:
Event Type: Warning
Event Source: MSExchangeTransportEvent Category: TransportServiceEvent ID: 12017
Description:A direct trust certificate will expire soon. Thumbprint:
2135A85FC400DF078D56A7A1EBB1E4330DD68596, hours remaining: 720
Event Type: Warning
Event Source: MSExchangeTransportEvent Category: TransportServiceEvent ID: 12015
Description:A direct trust certificate expired. Thumbprint:
2135A85FC400DF078D56A7A1EBB1E4330DD68596
Internal Transport - Determining Certificate
RTM use Certlib.ps1
SP1 use Get-TransportServer
Internal Transport
Error when Certificate Store cannot be found
Occurs when the Direct Trust certificate has been “forcibly” removed from the system (e.gMMC)
Agenda
Troubleshooting Mail flow
Local Delivery & Mail Submission: Architecture
Local Delivery & Mail Submission: Troubleshooting
Remote Delivery: Architecture
Troubleshooting Mail Flow
Troubleshooting Mail Queues
Mail Queues: Architecture
Mail Queues: Queues
Mail Queues: Troubleshooting
Troubleshooting Transport Certificates
Certificates and Transport: Overview
Certificate Selection
Troubleshooting STARTTLS
Internal Transport
Troubleshooting Direct Trust
SP1 Changes
218
Troubleshooting Direct Trust
Tools Used:Exchange Management Shell
TELNET
Protocol Logs
CertLib.ps1
1st Determine this is a direct trust issue
Troubleshooting Direct Trust - Direct Trust Issue?
[PS] C:\>Get-Queue 5 | fl
Identity : clt-e2k7\5DeliveryType : SmtpRelayWithinAdSiteToEdgeNextHopDomain : edgesync - northamerica to internetNextHopConnector : 4039e148-2af7-4889-9de8-6cf6f1b2716eStatus : RetryMessageCount : 1LastError : 451 4.4.0 Primary target IP address responded with:
454 4.7.0 Temporary authentication failure. Attempted failover to alternate host, but that did not succeed.
Either there are no alternate hosts, or delivery failed to all alternate hosts
.LastRetryTime : 9/19/2007 8:45:33 AMNextRetryTime : 9/19/2007 8:50:33 AMIsValid : TrueObjectState : Unchanged
Is ExchangeServer authentication enabled on the receive connector?
Make sure both Hub and Edge show X-ANONYMOUSTLS available
Troubleshooting Direct Trust
Confirm X-AnonymousTLS
Troubleshooting Direct Trust - Protocol Logging
2007-09-19T12:27:38.994Z,0,,10.0.200.200:25,*,,attempting to connect
2007-09-19T12:27:39.004Z,1,+,,
...
2007-09-19T12:27:39.014Z,17,>,X-ANONYMOUSTLS,
2007-09-19T12:27:39.024Z,18,<,220 2.0.0 SMTP server ready,
2007-09-19T12:27:39.024Z,19,*,,Sending certificate
2007-09-19T12:27:39.024Z,20,*,CN=clt-e2k7.fourthcoffee.com,Certificate subject
2007-09-19T12:27:39.024Z,21,*,"CN=LON-E2K7, DC=fourthcoffee, DC=com",Certificate issuer name
2007-09-19T12:27:39.024Z,22,*,610FCD9F000000000011,Certificate serial number
2007-09-19T12:27:39.024Z,23,*,91BFBDD870D8928018E22B736922411645218B85,Certificate thumbprint2007-09-19T12:27:39.024Z,24,*,clt-e2k7.fourthcoffee.com;clt-e2k7,Certificate alternate names
2007-09-19T12:27:51.292Z,25,*,,Received certificate
2007-09-19T12:27:51.292Z,26,*,3B1C9C4472B9ED9E9981262F48F164E4EDB02F0D,Certificate thumbprint
2007-09-19T12:27:51.292Z,27,*,SMTPSendEXCH50 SendRoutingHeaders SendForestHeaders SendOrganizationHeaders,Set Session Permissions
2007-09-19T12:27:51.292Z,28,*,,DirectTrust certificate
2007-09-19T12:27:51.292Z,29,*,CN=E2K7-EDGE1,Certificate subject
2007-09-19T12:27:51.292Z,30,*,CN=E2K7-EDGE1,Certificate issuer name
2007-09-19T12:27:51.292Z,31,*,126DA18D9F7FE69B48044E9D4985B894,Certificate serial number
2007-09-19T12:27:51.292Z,32,*,3B1C9C4472B9ED9E9981262F48F164E4EDB02F0D,Certificate thumbprint
2007-09-19T12:27:51.292Z,33,*,E2K7-EDGE1;E2K7-EDGE1.fourthcoffee.com,Certificate alternate names
2007-09-19T12:27:51.292Z,34,>,EHLO clt-e2k7.fourthcoffee.com,
2007-09-19T12:27:51.302Z,35,<,454 4.7.0 Temporary authentication failure,
2007-09-19T12:27:51.302Z,36,>,QUIT,
2007-09-19T12:27:51.302Z,37,-,,Remote
Troubleshooting Direct Trust
[PS] C:\>gettlscertfromad clt-e2k7Running on an Edge Server - pulling cert details from AdamSystem.DirectoryServices.DirectoryEntryRunning on an Edge Server - pulling cert details from Adam(&(objectclass=msExchExchangeServer)cn=clt-e2k7)Getting Prop
Thumbprint Subject---------- -------715F6840E3B4CC241CFDC9D912A5E8491B2BDE74 CN=clt-e2k7
[PS] C:\>gettlscertfromad e2k7-edge1Running on an Edge Server - pulling cert details from AdamSystem.DirectoryServices.DirectoryEntryRunning on an Edge Server - pulling cert details from Adam(&(objectclass=msExchExchangeServer)cn=e2k7-edge1)Getting Prop
Thumbprint Subject---------- -------3B1C9C4472B9ED9E9981262F48F164E4EDB02F0D CN=E2K7-EDGE1
Confirm Certificate Thumbprints:
Agenda
Troubleshooting Mail flow
Local Delivery & Mail Submission: Architecture
Local Delivery & Mail Submission: Troubleshooting
Remote Delivery: Architecture
Troubleshooting Mail Flow
Troubleshooting Mail Queues
Mail Queues: Architecture
Mail Queues: Queues
Mail Queues: Troubleshooting
Troubleshooting Transport Certificates
Certificates and Transport: Overview
Certificate Selection
Troubleshooting STARTTLS
Internal Transport
Troubleshooting Direct Trust
SP1 Changes
224
SP1 Changes
Certificate Logging
Get-ExchangeCertificate –Domain FOO.COM
Opportunistic TLS fallback
Direct Trust Certificate
Changing Terminology -“Internal Transport Certificate”
Updated anytime SMTP provided as service
Warnings upon update
Don’t have to re-subscribe w/update on Hub
No more 1037/2019 events
Attempts to fallback to alternate certificates if can’t be loaded
POP/IMAPSelection now prefers PKI
Supports Wildcards
SP1 Changes
EBDDE4C98F71840199B4256B9368F265A92DDB0C: Rejected. Unable to access the associated private key for the certificate.
Searching for a certificate that has one of the following FQDNs :
mail.fourthcoffee.com
60CF94F4522A25DFEBB734F4636FB1D1F77D819A: Is not valid for signing, dropping from consideration.
Considering certificate A7C7D7B88B767BAA6003A1F0DBE0DF0937210C68
A7C7D7B88B767BAA6003A1F0DBE0DF0937210C68: Rejected. Has a key size less than 1024 bits, dropping from consideration.
Considering certificate CA979162AC2854BBE389153D965358379F8CD43E
CA979162AC2854BBE389153D965358379F8CD43E: Selected. PKI issued certificate.
Logging Sample
Troubleshooting Mailbox Server(MBX)
Agenda
Databases
Public Folders
Recipient Management
Distribution Lists & Address Lists
Offline Address Book
Exchange Search
ExMON
ESE System
Transaction
Log File
715
525
415
841
10
Database Storage Files
Memory
7 15 25 4
17 8 3 1
10
.EDB Storage File
1 2 3 4
5 6 7 8
9 10 11 12
8 KB
Making Changes to the Database
Transaction
Log File
715
525
415
841
10
Database Storage FilesMemory
980
77
1001
7
1001
7
.EDB Storage File
1 2 3 4
5 6 7 8
1001
7
9 10 11 12
1001
Important JET Database Files
Three important file extensions to remember:EDB (Exchange database files)
LOG (Exchange Transaction logs)
CHK (Exchange transaction checkpoint files)
Checkpoint File
Exxnnnnn.log
Exx.chk
Transaction LogEntries Writtento the Database
Transaction LogEntries Not Yet Written to the
Database
Storage Groups
Set of all databases that share common log files
Separate instance of Jet
Up to 50 Storage groups per Server (Enterprise)
5 Databases per Storage Group but 50 maximum databases (Enterprise)
Recommendation is to have one database for one storage group
Utilities in 2007
No change compared to 2003!Eseutil command and options are the same but now can be also used on Hub and Edge transport databases
Isinteg command and options are the same
Troubleshooting Exchange Databases
For database problems all the troubleshooting techniques used in 2003 are still valid
Microsoft Confidential236
Continuous Replication flavors
LCR: Local Continuous Replication
CCR: Cluster Continuous Replication
SCR: Standby Continuous Replication
Microsoft Confidential237
Troubleshooting Continuous Replication
Exchange Management Console
Configuration
Management
Monitoring
Exchange Management ShellGet-StorageGroupCopyStatus
Test-ReplicationHealth
Performance countersMany counters available to monitor Continuous Replication
Microsoft Confidential238
Exchange 2007 SP1 Changes
Online Database Checksum (DBScan)
Addresses CCR scenario where backup occurs off passive (Active will never be completely scanned)
Opt in feature via Registry key
Set on a per server basis
Throttle parameter (unlimited by default)
Uses half of the Online Defrag Maintenance Window time
Will notify corruption via eventlog (-1018, -1022 etc)
Exchange 2007 SP1 Changes
Online Defrag Monitoring:New Perfmon Counter:MSExchange Database->Online Defrag Pages Freed/sec
Extended Event information:Event Type: Information
Event Source: ESE
Event Category: Online Defragmentation
Event ID: 703
Date: 6/20/2007
Time: 6:34:26 AM
User: N/A
Computer: DF-MBX-30
Description:
MSExchangeIS (19052) SG06: Online defragmentation has completed the resumed pass on database 'e:\MDB06\priv06.edb', freeing 42794 pages. This pass started on 6/16/2007 and ran for a total of 124919 seconds, requiring 7 invocations over 4 days. Since the database was created it has been fully defragmented 14 times over 73 days.
Exchange 2007 SP1 Changes
How to determine if OLD is running often enough?2 week rule of thumb or analyze serverAnalyze Perfmon log taken during OLD window
121:1
If Read:Freed ratio is greater than 100:1 then the OLD window can be reducedIf Read:Freed ratio is less than 50:1 then the OLD window should be increased
Why reduce?Increase backup windowReduce snapshot/block level differential sizes (DPM v2)Validate that Online Checksum/Page Zeroing can be introduced with current OLM window
Agenda
Databases
Public Folders
Recipient Management
Distribution Lists & Address Lists
Offline Address Book
Exchange Search
ExMON
Public Folder Replication
Mail based as in previous versions of ExchangeEnsure the public store has an email address
Check message tracking
Enable diagnostic logging
CCR cannot have a public store if there is more than one PF store
Force Public Folder replication
In the management consoleUpdate Hierarchy
Update content
Using the management shellUpdate-PublicFolderHierarchy
Update-PublicFolder
Public Folder Replication Storm
Public folder replication storm can still occur in Exchange 2007
Suspend-PublicFolderReplication
Stops all replication except hierarchy
Resume-PublicFolderReplication
Public Folder Referrals
Defines Client Public folder access
Routing Group based in Exchange 2000/2003
Active Directory Site based in Exchange 2007
Troubleshooting Referrals
Connection status in Outlook
Get-PublicFolderDatabase
UseCustomReferralServerList
CustomReferralServerList
Link cost (AD and RGC)
Agenda
Databases
Public Folders
Recipient Management
Distribution Lists & Address Lists
Offline Address Book
Exchange Search
ExMON
Recipient Management
Simplified Recipient Provisioning for the Exchange Administrator
Support for “Split Permissions” within a single forest
Ability to delegate Recipient management to a lower level administrator
Ability to create Active Directory object and mail- or mailbox-enable it
Instant-on recipients – no need to wait or “kick” the RUS to stamp objects
Rich “filtering” support – includes domain- and forest-wide scoping
Allows administrators to see only the objects relevant to them
New recipient types plus clear distinction of all recipient types
Conference Room and Equipment Mailbox (Resource Mailbox)
Policy support for select mailbox settings
Ability to apply the same settings to all recipients associated with a policy
Unified Messaging, Messaging Records Management, and ActiveSync
Recipient Policies still exist but are now called E-Mail Address Policies
Page 253
Working with Recipients
Recipients are primarily mailbox-enabled Active Directory users
Recipients are managed through the Exchange Management Console or the Exchange Management Shell
Active Directory Users and Computers (ADUC) is no longer extended for management of Exchange recipients
User AD properties relevant for the Global Address List can be managed through the Recipient Configuration container in Exchange Management Console
Active Directory User accounts can be created from within the Exchange Management Console when they are mailbox-enabled
Page 255
Working with Recipients and ADUC
Active Directory Users and Computers (ADUC) is no longer extended to manage Exchange recipients
It is not supported to mailbox-enable user accounts using ADUC when the mailboxes will be housed on Exchange 2007 servers.
If there is an Exchange Server 2003 RUS server operational, the ADUC mailbox operation will succeed, so the mailbox will be able to send and receive messages
Mailbox is considered legacy and certain features or actions, or properties will be blocked
Set-Mailbox -ApplyMandatoryProperties
Page 256
Page 259
Scoping
Recipient Configuration Center supports domain- andforest-wide scoping
Ability to specify which DC Console should connect to
Scope is configurable, even down to OU
$AdminSessionADSettings session variable (in shell)
Domain Scope is default behavior
Determined by domain of which the Server is a member:
Only recipients (e.g., redmond\evand) in selected domain can be found
Referenced recipients (e.g., Membership, Delegate, Owner, etc.) are exempt
Reduces issues related to replication
Forest Scope can display and find all recipients within the forest
Provides a complete view of the GAL
Enable/Disable vs. New/Remove
Enable/Disable
Adds or removes Exchange attributes from existing Active Directory® objects
Enable – adds attributes to an existing Active Directory object –mail-enabled or mailbox-enabled
Disable – removes attributes returning Active Directory object to non-Exchange state
StoreMailbox in MDB will fall under mailbox retention and will eventually be purged
New/Remove*Creates or deletes Active Directory® objects plus adds and removes Exchange attributes in one step
New – creates Active Directory object and mail-enables or mailbox-enables the objectDefault Remove – removes Active Directory object. StoreMailbox in MDB will fall under mailbox retention and will eventually be purged-Permanent: removes Active Directory object and StoreMailboxin MDB will be purged immediately (shell only)
* Must have Account Operator privileges
Page 262
Email Address Policies
Created pre-canned filters to simplify definition and usage forcommon cases
All Recipient Types, Users with Mailboxes, Resource Mailboxes, Mail-Enabled Contacts, and Mail-Enabled Groups
Conditions Supported: State or Province, Department, Company, and Custom Attributes
Ability to schedule the creation and application of Email Address Policies foroff-hour execution when using EMC
RUS as a service no longer needed, resulting in reduced system processing demand
Mailbox Manager functionality separated from EAPs
Replaced by Messaging Records Management functionality
Advanced or Non-Mainline (Shell Only)
Custom Filters - will be visible, but not editable, in GUI
Page 263
Managing Mailboxes
Well-known functionality are still there
New mailbox
Move mailbox
Delete mailbox
Change Mailbox properties
Page 265
New Mailbox Management tasks
Statistics
Get-LogonStatistics
Get-MailboxStatistics
Get-MailboxFolderStatistics
Troubleshooting Mailbox access
Test-MapiConnectivity
Outlook logging
831053 How to turn on the Enable Mail Logging option for troubleshooting in Outlook 2003 and Outlook 2007
Network
Take network trace
Reproduce the problem locally
Moving Mailboxes
You can use the Exchange Management Console or the Exchange Management Shell to move mailboxes
You can move mailboxes across mailbox databases, across servers, across domains, across Active Directory sites and across forests
You can also move mailboxes among different versions of Microsoft Exchange Server (2000/2003/2007 only)
Move mailbox is more resilient (Pre-Validation)
Exchange Management Shell Command: move-mailbox
More options available
Note: You cannot use the Exchange Management Console to move mailboxes across forests. You must use the Exchange Management Shell instead.
Page 269
Troubleshooting Mailbox Move
Email Address Enforcement
IgnoreRuleLimitErrors cmdlet option
Damaged or corrupted messages
BadItemLimit cmdlet option
Skip errors validation from EMC Move-MailboxWizard
MfcMapi
Isinteg
Exmerge Replacement Need
Exmerge is not shipped with Exchange Server 2007
The Move-Mailbox, Export-Mailbox, and Restore-Mailbox tasks are implemented to cover many of the scenarios where ExMerge is used with Exchange Server 2003
Export to PST is possible in SP1
Page 274
ExMerge Replacement cmdlets
Export-MailboxExport mailbox content to another mailbox or PST
Must run 32-bits if exporting to PST
Must have Outlook installed
Can filter content
Can delete source message
Will export dumpster
Import-MailboxImports from PST
Must run 32-bits console
Restore-Mailbox
Microsoft Confidential278
LAB 7 Troubleshooting MAPI access
LAB 8 Using MFCMAPI
Agenda
Databases
Public Folders
Recipient Management
Distribution Lists & Address Lists
Offline Address Book
Exchange Search
ExMON
Distribution List Types
Mail-enabled Universal Distribution Group
Mail-enabled Universal Security Group
Mail-enabled Non-Universal Group
Dynamic Distribution Group
Automatic Group conversion
Users can select Universal Distribution Group to set permissions on folders
Exchange will automatically convert the group to security
Can potentially growth user security token
Can be disabledSet msExchDisableUDGConversion through ADSIEdit
Interaction with Exchange 2003
Dynamic Distribution groups created in Exchange 2003 must beupgraded to be modified in Exchange 2007
Get-DynamicDistributionGroup | Format-List Name,*RecipientFilter*,ExchangeVersion
If RecipientFilterType is "Legacy“ and ExchangeVersion is "0.0 (6.5.6500.0)” Set-DynamicDistributionGroup –recipientfilter {... } –forceupgrade$true
Exchange 2007 Distribution List can only use Universal groupscope
Common Issues
Unable to send to the Distribution Group from users external tothe Organization
When “Require that all senders are auhenticated “ flag is set on DLProperties
To solve the issue run: Set-DistributionList –RequireSenderAuthenticationEnabled $true
Unable to view the Distribution Group in EMC
When the group scope is Global or Domain Local
To solve this issue change the group scope to Universal scope
Default Address Lists
Default Global Address List
All Contacts
All Groups
All Rooms
Public Folders
All Users
Populating the Address Lists
In Exchange 2007 there is no Recipient Update Service (RUS)
PreCanned Filters
In order to update an Address List you have to run the cmdlet Update-AddressList
Update-AddressList can be scheduled to run using Exchange Management Shell
Issues
Unable to Edit the Address List Properties (Address List must beupgraded)
If ALs created by using Exchange 2003
Upgrade them to Exchange 2007 to use OPATH filtersSet-GlobalAddressList "Default Global Address List" -RecipientFilter {(Alias -ne $null -and (ObjectClass -eq 'user' -or ObjectClass -eq 'contact' -or ObjectClass -eq 'msExchSystemMailbox' -or ObjectClass -eq'msExchDynamicDistributionList' -or ObjectClass -eq 'group' -or ObjectClass-eq 'publicFolder'))}
Address List are not updated after modifying them
Exchange 2007 has no RUS to update them
Address Lists must be updated using EMC or EMS (Update-AddressList)
RUS API can be troubleshooted via :
Get-EventLogLevel MSExchangeAL | Set-EventLogLevel -Expert
Agenda
Databases
Public Folders
Recipient Management
Distribution Lists & Address Lists
Offline Address Book
Exchange Search
ExMON
What is the Offline Address Book?
An Offline copy of the Global Address List
Used by Outlook clients in Offline mode or Cache mode
Several versions appeared over time
Version 2 appeared in Exchange 5.5
For clients Outlook 98 and later
Version 3 Appeared in Exchange 2003
For clients Outlook 2003 and later
Version 4 Appeared in Exchange 2003 SP2
For clients Outlook 2003 SP2 and later
Microsoft Confidential292
Page 293
Offline Address Book Overview
Exchange 2007 introduces a new mechanism for distributing the OAB that does not require Public Folders
HTTP(S) and Background Intelligent Transfer Service (BITS) can be used by Outlook 2007 clients
Advantages of this new method are:
Support for more concurrent clients
Reduction in bandwidth usage
More control over the distribution points
The web distribution is only available for OABv4
Legacy Clients will still need to utilize public folders for OAB obtainment
Outlook 2007 clients can utilize the web distribution for obtaining the OAB
3/24/2009 | Page 294
Offline Address Book generation
The Offline Address Book (OAB) is generated as usual by the OABgen component on a Mailbox Server
Files are published to \\mbxserver\ExchangeOAB share and to the public store if available
On all Client Access servers, an OAB virtual directory is created to serve the OAB
The Exchange File Distribution Service that runs on the CAS servers is responsible to getting the OAB content from the OABGen server
The virtual directory points to the directory %programfiles%\microsoft\exchange server\ClientAccess\OAB
In that directory, the different OABs are stored per <guid>
The .lzx files contains the OAB data in V4 format
The oab.xml contains metadata for Outlook 2007
Outlook 2007 is configured to retrieve the OAB via the OAB URL that is obtained through AutoDiscover. Otherwise it will download OAB from public folders like all other legacy clients
Offline Address Book size and network bandwidth usage
Started to become an issue with Outlook cache mode deployments
No limit for Public Folder connections
OAB throttling to control network bandwidth usage
Outlook Random Full OAB Request Timer
Key: HKCU\Software\Microsoft\Exchange\Exchange Provider
DWORD: Max Full OAB Download Wait
Value: Integer >=1
Microsoft Confidential295
OAB V4 Improvements
OAB V4 is more compressed
Binpatch technology and LZX compression method used
Rebuild needs are reduced
Indexes generated by Outlook Client
Limited property sizes
Web distribution optimizes network bandwidth usage
Microsoft Confidential297
Page 299
Offline Address Book Web Distribution Scenario
Scenario
Corporate Headquarters (London)
User A
Remote Office
(Sao Paulo)
User B
User C
The Internet
CAS Server
Mailbox ServerFast Connection
Slow Link
Legend
Outlook
Web distribution self healing
Microsoft Confidential302
OAB generated files are kept within the System Attendant mailbox
Deleted files from the mailbox role OAB share will be copied back
Deleted files from the CAS web virtual directory will be copied back from the Mailbox OAB share
Offline Address Book Public folder distribution
Microsoft Confidential303
Outlook clients will connect through RPC to the public folder server holding a replica of the OAB
To reduce bandwidth usage you should:
Make sure to use OAB V4
Replicate the OAB on a public folder in every Active Directory site holding a Mailbox role
Or create an OAB per site and assign the mailbox stores to the local OAB
Don’t forget OAB Threshold registry setting
OAB Version used
Outlook 2003 SP2 and Outlook 2007 can use V4
Will failback on previous version if not available
Ensure that Version4 of the OAB is enabled
Get-OfflineAddressBook | fl Name,Server,Versions
Set-OfflineAddressBook –Versions Version4
If the profile is ANSI, OAB V2 will be used
Mainly for profiles linked to mailboxes moved from Exchange 5.5
Deploy GPO to force Profile conversion
Set registry key to force OAB V4 use
Key: HKCU \Software\Microsoft\Exchange\Exchange Provider
DWORD: OAB v4 Only
Value: 1
Microsoft Confidential305
Troubleshooting OAB using diagnostic logging
Set Diagnostic level
Set-EventLogLevel -Identity “MSExchangeSA\OAL Generator” -Level Expert
Read event logs
Using the event viewer
Using Powershell
Get-EventLog Application | Where {$_.Category -eq"OAL Generator"}
Microsoft Confidential309
Offline Address Book Integrity Checker (OabInteg)
Tool to simulate
Client connection to download OAB files from public folder store
Does not yet test web distribution (should be available soon)
OAB generation process
Downloadable from InternetYou can download OABInteg from here: http://gotdotnet.com/Community/UserSamples/Download.aspx?SampleGuid=A2338E73-F521-4071-9B1D-AAF49C346ACD
If run from the server install CDO 1.2.1 to test MAPI access. Downloadable from:
http://www.microsoft.com/downloads/details.aspx?FamilyID=e17e7f31-079a-43a9-bff2-0a110307611e&DisplayLang=en
Microsoft Confidential312
OAB Generation Errors
Exchange server configured to generate the OAB
By default it is first Exchange Server in org which isExchange 2003 in mixed modes
In mixed mode:
Move OAB from Exchange 2003 server to Exchange 2007 server
Local replicas of OAB on Exchange 2007 server should be successfully replicated
All mailbox stores on Exchange 2007 server under Client Settings tab should have Default Offline Address Book associated
Microsoft Confidential314
OAB Generation on CCR Clusters
CCR cluster
Only one node is generating OAB
When the node becomes passive OAB is not updated
Logs error event 9395
How to Fix:HKLM\System\CurrentControlSet\Services\MSExchangeSA\Parameters\Server-Name\EnableOabGenOnThisNode ="ThisNodeName"
Microsoft Confidential316
OAB Download troubleshooting
Outlook send receive errors are saved in the “sync issues” folder
Use Err.exe to interpret the error
Enable Outlook logging
831053 How to turn on the Enable Mail Logging option for troubleshooting in Outlook 2003 and Outlook 2007
http://support.microsoft.com/default.aspx?scid=kb;EN-US;831053
Check which OAB to download
Configured either on the mailbox store or on the recipient itself
Check Application log
Take a network trace
Microsoft Confidential318
OAB Download from public folder troubleshooting
Has the OAB been generated on public folder?Is the OAB public folder enabled?
Is the version we want to download available?
Verify Application log on the OAB generator server
Look in the OAB public folder
Use OABInteg
Is the OAB public folder reachable by Outlook?
Verify where are the replicas of the OAB we should download
Make sure the public store holding the nearest replica is reachable
Check public folder replication or referrals
Look in the Outlook connection status if Outlook did effectively connect to the server holding the replica
Throttling might be preventing download
Microsoft Confidential320
OAB Download from Web distribution troubleshooting
Make sure OAB is configured to generate V4Only V4 are web distribution enabled
Make sure the Outlook profile is UnicodeAnsi profile wouldn’t be able to download V4
Check if OAB has been generated on the Mailbox serverWeb distribution won’t be generated if system attendant can’t logon to its mailbox
Check if it has been replicated to the OAB virtual directories
Exchange file distribution service will copy it from the mailbox server to the CAS
Verify Outlook auto configuration (See CAS module)
Microsoft Confidential321
Troubleshooting out of date OAB
Verify by connecting Online that inconsistency is only on OAB
Entries not stamped by Recipient Update API
Update-Recipient
Force an update and check event logUpdate-OfflineAddressBook
Out of date only for old Outlook clients
Those clients will use old OAB version. Might be an issue when Exchange 2007 is introduced in an existing organization
Active Directory replication
The Global Catalog used to generate the OAB may have directory replication issues. Use tools like replmon or repamin to verify
Microsoft Confidential324
Microsoft Confidential327
LAB 9 Troubleshooting OAB
Agenda
Databases
Public Folders
Recipient Management
Distribution Lists & Address Lists
Offline Address Book
Exchange Search
ExMON
Exchange Search
Understanding Exchange Search
Difference between Exchange Search and store search
Unexpected results scenarios
Troubleshooting Exchange Search
Understanding Exchange Search
Microsoft Exchange Server 2007 Search is a feature that allows you to quickly search text in messages through the use of pre-built indexes
Indexes occupy approximately 5 percent of the total mailbox database size
Kept separately in same location as database files
Page 331
Used by OWA and Outlook online mode
Outlook cached mode uses new client-side search – Windows Desktop Search
Instant Search goes through attachments in Outlook
Can be extended to use any filter in Windows
Performance Enhancements
Page 332
Outlook in online mode
Exchange Server 2007 Search Indexer and advanced find in Outlook 2007
Faster indexing than Exchange Server 2003 and Exchange Server 2000
New messages indexed in under a minute
Small storage tax (~5%) for indexes
Indexes/searches message bodies and attachments
Uses any filter installed in Windows
Can install new filters later
Outlook in Cached Exchange Mode
On Windows XP, Outlook uses Windows Desktop Search
On Windows Vista, Outlook uses Vista‟s built-in search engine
Performance Improvements
Mailbox
Understanding Exchange Search
Index
MS Information Store
Service
Exchange Search Service
Notificatio
n
Update
Difference between Exchange Search and store search
Exchange Search Exchange Store Search
Faster Slower
Based on words Based on bytes stream
Search attachments* Cannot Search attachments
Uses index to search Uses serial scans
Not case sensitive Case Sensitive
Doesn‟t support MAPI
restrictionsSupport MAPI restrictions
* Attachments types that are supported by the installed filters
Unexpected results scenarios
Documents that are encrypted with the Digital Rights Management feature will not be indexed.
For attachments that do not have associated filters, the attachment will not be indexed, but the e-mail message will be indexed.
Advanced search grammar (for instance, typing "From:xyz" in the basic search bar searches the from: property for the string "xyz) is supported only when Instant Search is enabled. Instant Search requires that Windows Desktop Search 3.0 is installed.
Troubleshooting Exchange Search
Step 1• Is the MSExchangeSearch service started?
Step 2
• Is the IndexEnabled parameter set to true
• Get-MailboxDatabase |ft Name,IndexEnabled
Step 3
• Has the Exchange database been crawled?
• MSExchange Search Indices performance object=0
Step 4• Run the Test-ExchangeSearch
Step 5
• Check Event Viewer
• Source: MSExchangeSearch Indexer
Step 6• restart the Microsoft Search
Test-ExchangeSearch
The Test-ExchangeSearch cmdlet creates a message and attachment that only the Microsoft Exchange search can find. Unless a mailbox is specified in the Identity parameter, the message is stored in the System Attendant mailbox. The command waits for the message to be indexed and then searches for the content. The command reports success if the message content is found. The command reports failure if the content is not found after the interval set in the IndexingTimeout parameter has elapsed.
To run the Test-ExchangeSearch cmdlet, the account you use must be delegated the following:
1. Exchange Recipient Administrator role-and-
2. Exchange Server Administrator role and local Administrators group for the target server
How to rebuild the search index
Programmatically: use the ResetSearchIndex.ps1
Manually stop the service and deleting the file
GetDatabaseForSearchIndex.ps1 When the index directory files are provided, this script returns the associated mailbox database names.
GetSearchIndexForDatabase.ps1 This script returns index directories for the specified mailbox database names.
Microsoft Confidential339
LAB 10 Troubleshooting Search
Agenda
Databases
Public Folders
Recipient Management
Distribution Lists & Address Lists
Offline Address Book
Exchange Search
ExMON
What is Exmon?
Originally developed by Microsoft to understand user load on servers
Shows per user activity in details
Allows to track down high users
Introducing ExMon
Administrators can view the following using ExMon:
IP addresses used by clients
Microsoft Office Outlook® versions and mode (Cached Exchange Mode versus classic online)
Outlook client-side monitoring data
CPU usage
Server-side processor latency
Total latency (network and processing)
Network bytes
In Exchange 2007 works for all mailbox access
Microsoft Confidential342
When to Use ExMon
Some Outlook users are complaining about latencies regarding mailbox access
RPC Average Latency is high
Want to know what Outlook versions are really in use
Want to find high RPC activity users
Are they in cache mode?
Want to know who is working among connected users
Determine usage pattern on healthy systems
Microsoft Confidential343
Collecting ExMon Data
ExMon data can be collected in 3 ways
Live from the ExMon User Interface
Command Line
Perfmon collection
Command line or perfmon are recommended
The 'Live' mode constantly rolls, does not save data
Can use server CPU to process data
Command Line and Perfmon accommodates large files and ability to script and control
Larger files give much more insight including
Better aggregate statistics
Some data (like process name) is only traced on MAPI logon
ExMon data should be collected to local disks only
Consumes roughly 1MB/hour for every RPC Operation/sec
Viewing ExMon Data
Must be viewed on same OS or higher as collected on
Windows 2003 Server required to view data from Windows 2003 Server
Large files can take a long time to open, use CPU
Saving your work…
Command Line can save any 'By …' without displaying UI
File->Save will save all 'By …' views in one .csv
By Event (for a given user) can be saved only in UI
The Save Icon on the toolbar instructs ExMon to save ETL files captured during ‘Live Capture’
Analyzing Exmon Data
Know your environment
Establish baseline to compare
Detect RPC Average Latency peak using the performance wizard
Main Exmon points
CPU Time
Server Latency
Client Latency
Foreground Client Latency
Network Bytes
Basic Principles
Focus on the most expensive Users or Operations (unless you are troubleshooting a particular user)
Statistics are best for expensive operations or LOTS of inexpensive one
Look for problem to repeat in ExMon, then tackle
Longer captures are better than short ones
Expect some expensive operations to happen
Full sync of an OST
Occasional searches and sorts
Trick is to find the ones that happen frequently or really hurt
When looking at an individual user
Look for patterns of repetition
Compare to 'normal behavior'
© 2008 Microsoft Corporation. All rights reserved.
Information in this document, including URL and other Internet Web site references, is subject to change without notice. Unless otherwise noted, the example companies, organizations, products, domain names, e-mail addresses, logos, people, places, and events depicted herein are fictitious, and no association with any real company, organization, product, domain name, e-mail address, logo, person, place, or event is intended or should be inferred. Complying with all applicable copyright laws is the responsibility of the user. Without limiting the rights under copyright, no part of this document may be reproduced, stored in or introduced into a retrieval system, or transmitted in any form or by any means (electronic, mechanical, photocopying, recording, or otherwise), or for any purpose, without the express written permission of Microsoft Corporation.
For more information see Microsoft Copyright Permissions at http://www.microsoft.com/permission
Microsoft may have patents, patent applications, trademarks, copyrights, or other intellectual property rights covering subject matter in this document. Except as expressly provided in any written license agreement from Microsoft, the furnishing of this document does not give you any license to these patents, trademarks, copyrights, or other intellectual property.
The Microsoft company name and Microsoft products mentioned herein may be either registered trademarks or trademarks of Microsoft Corporation in the United States and/or other countries. The names of actual companies and products mentioned herein may be the trademarks of their respective owners.
THIS DOCUMENT IS FOR INFORMATIONAL AND TRAINING PURPOSES ONLY AND IS PROVIDED "AS IS" WITHOUT WARRANTY OF ANY KIND, WHETHER EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE IMPLIED WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE, AND NON-INFRINGEMENT.
Terms of Use