Welcome message from author
This document is posted to help you gain knowledge. Please leave a comment to let me know what you think about it! Share it to your friends and learn new things together.
Transcript
How to manage and troubleshoot Search: A practical guide Brian Pendergrass and Jon WaitePremier Field EngineersMicrosoft
SPC375
Who is this session for?SharePoint IT professionals who support/manage Search environments300 level contentDiscussion of Search internals
Ask the ExpertsCome find us to talk more about Search
The decoder ring…SSA QPC SQ&SSSearch Service Application Query Processing Component Search Query & Site Settings
Juno/MarsInternal naming conventions for Search that appear in ULS for various activities
WCF (Windows Communication Foundation)A framework for building service-oriented applications where clients communicate with services through defined endpoints
WSDL (Web Services Description Language)Describes the complete contract for application communication
AAMs( Alternate Access Mappings)Enable a Web application that receives a request for an internal URL to return pages that contain links to the public URL for the zone
SharePoint 2013 Search Architecture
Unit of Scale / Role boundary
Admin
mssearch.exe noderunner.exe xnoderunner.exe
Query Processing(IMS) WFE
noderunner.exe
Search
Admin
Analysis Engine (APC)
CRAWL
STORECrawl
Store
Searc
h Q
uery
an
d S
ite S
ett
ing
s
(
wit
h s
earc
hse
rvic
e.s
vc
En
d
Poin
ts)
Analytics Reporting
SS
A P
roxy
Query
OM
Serv
er-
side
UI
Query
CSO
M
Links StoreUsage DB
SharePoint Crawler(Crawl)
Content Processing Search core
FAST search core
Custom Enrichment Web Service
mssdmn.exe
mssdmn.exe
mssdmn.exe
mssdmn.exe
CSS (Load
Balancer)
…
Processing Flows
Index Router
Journal
\\srv\gthrsvc network
share
Content Plug-In API
Content
Protocol Handlers• sts4[s]
(SharePoint)• sps3[s] (People)• http[s] (Web)• file (File Share)• bdc (Business)
Rele
vanc
y E
ngin
e
Que
ry F
low
s
IMS Query Pool
…
Result Mixer
Mars Lookup Operator
(MLO)
even
ts
w3wp.exe
Content Router
mssearch.exe w3wp.exe
w3wp.exe
noderunner.exe
UX
Crawling and Feeding Chain
Index Core
Query Chain
Analytics Service
noderunner.exe
Five key takeaways from this session…Application Server Administration Service Timer JobSynchronizes search instances across the farm
Crawl state lives in the databaseMore precise state can be found in the MSSCrawlHistory/MSSCrawlComponentsState tables
Follow a crawled document via ULSUse VerboseEx ULS to track from Crawler to CPC to Index
Follow a query via ULSUse Merge-SPLogfile, Query starts at the Web Front End
Crawl the default zoneDon’t use server name mappings for SharePoint content
Troubleshooting Crawling and Feeding
Problem: My Crawl is stuck!Where to look:Check Central Admin Search Topology page for errorsCheck Timer Service/Timer JobsCheck ULS logs on Crawler for errorsCheck Search databases
MSSCrawlHistoryWhat is the crawl status? Requested status?
MSSCrawlComponentStateWhat are the component states for your crawl?
MSSCrawlQueueAre items being added/removed?
MSSCrawlURLAre items marked with this crawl ID being changed?
Are there blocking/blocked queries on the databases?
Central Admin Search TopologyCheck Central Admin Search Topology page for errors Ensure the components are online (green checkmark)
Check Timer Service/Timer JobsApplication Server Administration Service Timer Job Ensure this job is running and Synchronize is succeeding by examining ULS
Message: Name=Timer Job job-application-server-admin-serviceMessage: synchronizing search service instance
Message: Done synchronizing SearchServiceInstance.Message: Leaving Monitored Scope (Timer Job job-application-server-admin-service). Execution Time=XXXX
ULS LogsExample:The application cannot be initialized.Context: Application 'Search_Service_Application'
Details:The gatherer is shutting down (0x80040d23)
ERRORApplication Server Administration job failed for service instance Microsoft.Office.Server.Search.Administration.SearchServiceInstance (ee54f9c5-2b43-4466-9e21-483389331abb).
Reason: The gatherer is shutting down.
Technical Support Details:System.Runtime.InteropServices.COMException (0x80040D23): The gatherer is shutting down. at Microsoft.Office.Server.Search.Administration.MSSITLB.IGatherManagerAdmin3.get_ConfigurationVersion() at Microsoft.Office.Server.Search.Administration.Gatherer.ProvisionGlobalProperties() at Microsoft.Office.Server.Search.Administration.SearchServiceInstance.Synchronize() at Microsoft.Office.Server.Administration.ApplicationServerJob.ProvisionLocalSharedServiceInstances(Boolean isAdministrationServiceJob)
ULS LogsCheck ULS logs on Crawler for errorsIf Application Server Administration Service Timer Job isn’t running, you may see no entries at all.
SSP Job Control Service instance Status = onlineHidden “Service on Server”PowerShell:
$sspjob = (Get-SPFarm).Servers[“<servername>"].ServiceInstances | where {$_.typename -eq "SSP Job Control Service"}
The SharePoint Crawl is stateful with transitional “ing” states (e.g. starting, completing, stopping, pausing) and corresponding rest states (e.g. started, completed,
stopped, paused)
Crawl state lives in the database
Status SubStatus
This combination means… Crawl Component Crawl DB
0 0 New crawl, requesting start x1 1 Starting, Add Start Address(es) x1 2 Starting, waiting on Crawl Component(s) x4 1 Crawling4 2 Crawling, Unvisited to Queue4 3 Crawling, Delete Unvisited x4 4 Crawling, Wait for All Databases x5 0 Failed to Start (e.g. Another Crawl Already Running)7 Resuming x8 1 Pausing, Waiting on Crawl Component(s) to Pause x8 2 Pausing, Complete Pause x9 Paused
11 0 Completed12 Stopped13 1 Stopping, Waiting on Crawl Component(s) to Stop x13 2 Stopping, Complete Stop x14 1 Completing, Waiting on Crawl Component(s) to Complete x14 2 Completing x14 4 Completing, Get Deletes Pending x
Translating MSSCrawlHistory Status
Source:[SSA].[dbo].[proc_MSS_CrawlAdmin]
For Crawls Waiting on Crawl Components…SELECT hist.CrawlId, hist.CrawlType, hist.ContentSourceId, hist.StartTime, hist.Status, hist.SubStatus, hist.Request, ccState.ComponentID, ccState.Status as ccStatus, ccState.SuspendedCount FROM [SSA].[dbo].[MSSCrawlHistory] AS hist WITH (nolock) INNER JOIN [SSA].[dbo].[MSSCrawlComponentsState] AS ccState ON hist.CrawlId = ccState.CrawlId WHERE hist.Status Not In (5,11,12)and hist.crawlid > 2
Are there blocking queries on the databases?Check with the DBACheck the SQL Server Blocked Queries ReportDoes SQL have adequate drive space for database/logs?
If you do find a blocked query, you need to understand why/what it’s doingSearch will often simply re-issue the same blocking query as soon as it failsThis may mean updating statistics so SQL can choose a better execution plan
Yes, what do we do?
Crawling/Gathering
Crawl Component
Content Processin
g Componen
t
Follow a crawled document via ULS
Content
\\srv\gthrsvc network
share
blob
Crawl Store
AF7YN
Connector
DS86
AI3EF
Index Component
AJPND
E5G1
Feeding/Processing
Troubleshooting Queries
Searc
h
Ind
ex
Core
Following the path of a Query…
User quer
y
Web01
JunoSendRequest MarsLookupOperator
[HTTP] EndPoint: SearchService.svc
SQ&SSSearch Query
& Site Settings
WCFSendRequest
For a query, a single Correlation ID spans across components and servers
QPCQuery Processing
Component
[net.tcp] EndPoint ImsQueryInternal
http://web01/results.aspx?k=tps
Query: Sorry, something went wrong.
SharePoint 2013 SharePoint 2010
Demo1. Run a query …and watch if fail
with “Sorry, something went wrong”
2. Use the Correlation ID to filter ULS events across all applicable servers Merge-SPLogFile -Path $target -Correlation $id
A case of something going wrong…
SharePoint Foundation Topology e5mc MediumWcfSendRequest: RemoteAddress: 'http://qpcServer:32843/-GUID-of-the-SSA-/SearchService.svc' Channel: 'Microsoft.Office.Server.Search.Administration.ISearchServiceApplication' Action: 'http://tempuri.org/ISearchQueryServiceApplication/Execute'
SharePoint Server Search Query dka5 HighSearchServiceApplicationProxy::Execute--Error occured: System.ServiceModel.ServerTooBusyException: The HTTP service located at http://qpcServer:32843/-GUID-of-the-SSA-/SearchService.svc is unavailable. This could be because the service is too busy or because no endpoint was found listening at the specified address. Please ensure that the address is correct and try accessing the service again later. ---> System.Net.WebException: The remote server returned an error: (503) Server Unavailable. at System.Net.HttpWebRequest.GetResponse()
User quer
y
Web01
SQ&SSSearch Query
& Site Settings
WCFSendRequest SQ&SSSearch Query
& Site Settings
Common networking issues
Ports and ProtocolsPorts used by the search index component — TCP 16500-16519 (intra-farm only)Ports required for Windows Communication Foundation communication — TCP 808Ports required for communication between Web servers and service applications (the default is HTTP):• HTTP binding: TCP 32843• HTTPS binding: TCP 32844
Disable TCP Chimney/Task Offloadnetsh int tcp show global | findstr Chimney
netsh int ip show global | findstr "Task Offload"
Important event in ULS for Query PerfSearch Query Processing aizgm MediumMicrosoft.Office.Server.Search.Query.Pipeline.Executors.QueryPipelineFlowExecutor : FlowExecutor done: SearchApplication=cb4954e3-e4ad-40bd-b0e2-1b1db290c668. ParentFlow=Microsoft.ProductivitySearchFlow SubFlowTimings: QueryRuleConditionMatching=468 QueryTransformer=31 QueryRouter=1093 ResultsMixer=0 LayoutsSelector=0 QueryLogging=15
Search Search Component ajkph HighMicrosoft.Ceres.SearchCore.Query.MarsLookupComponent.LookupService.QueryClient.QueryExecutor: ExecuteQuery timings for correlation: dad1789c-7d1d-8016-3b36-b562c4fdae55, task dispatch 0 ms, blocked waiting 254 ms, total hits: 84, with dupes: 92, bytes received: 33238, 1 tasks: (cell: I.0.0 at IndexComponent1, total task time: 79 ms, query push queue: 174 ms, send to index call duration: 20 ms, total in transit: 63 ms, index node execute queue: 0 ms, query lookup: 7 ms, docsum lookup: 9 ms)
Where’s this SQ&SS you speak of?
Search Query and Site Settings Service
Note: The SQ&SS will be automatically started on all servers with a QPC, but should not be enabled on any other server
Search Query and Site Settings ServiceGet-SPEnterpriseSearchQueryAndSiteSettingsServiceInstanceTypeName Status Id-------- ------ --Search Query and Site Settings Service Disabled 838d344e-cbe2-4045-9350-1f4d19c2413eSearch Query and Site Settings Service Disabled 0110417b-163b-41cb-9676-63cb80d65b52Search Query and Site Settings Service Disabled 6ec6ea28-c732-48dd-ab83-5458a577a300Search Query and Site Settings Service Disabled d301abbf-4a74-4897-933c-3a4041c40a0fSearch Query and Site Settings Service Online 65708c36-66fb-41b5-b622-6a97c4e0aa26
foreach ($pt in $SSA.EndPoints) { $pt.ListenUris.AbsoluteUri }http://jwspc14spc5:32843/cb4954e3e4ad40bdb0e21b1db290c668/SearchService.svchttps://jwspc14spc5:32844/cb4954e3e4ad40bdb0e21b1db290c668/SearchService.svc
Search Load Balancing IMS/Query Flows
User quer
y
Web01
WCFSen
dReque
st Searc
h
Ind
ex
Core
QPCQuery Processing
Component
JunoSendRequest MarsLookupOperatorSQ&SSSearch Query
& Site Settings
QPCQuery Processing
Component
SQ&SSSearch Query
& Site Settings
Farm Load Balancing WCF Service EndPoints
Farm: Load Balancing the SSA EndPoint
Search Load Balancing IMS/Query Flows
User quer
y
Web01WCFSendRequest
Searc
h
Ind
ex
Core
QPCQuery Processing
Component
JunoSendRequest
MarsLookupOperatorSQ&SSSearch Query
& Site Settings
QPCQuery Processing
Component
SQ&SSSearch Query
& Site Settings
Farm Load Balancing WCF Service EndPoints
Farm: Load Balancing the SSA EndPoint
Farm Topology: Error Handling
SharePoint Foundation Topology 8313 CriticalA failure was reported when trying to invoke a service application: EndpointFailure Process Name: OWSTIMER Process ID: 12345 AppDomain Name: DefaultDomain AppDomain ID: 1 Service Application Uri: urn:schemas-microsoft-com:sharepoint:service:144f1459cc294525bf691… #authority=urn:uuid…&authority=https://someserver:32844/Topology/topology.svcActive Endpoints: 2 #the total number of known EndPoints for this Service Application Failed Endpoints:1 Affected Endpoint: http://qpcServer:32843/-GUID-of-the-SSA-/SearchService.svc
User quer
y
Web01
WCFSendRequest
SQ&SSSearch Query
& Site Settings
SQ&SSSearch Query
& Site Settings
SQ&SSSearch Query
& Site Settings
The Farm takes this failed EndPoint out of rotation for 10
min
The impact of Alternate Access Mappings
Auto-magically mapped to current zone
User quer
y
Web01 Query Results:
• http://usersUrl/foo/DispForm.aspx?ID=3• http://usersUrl/TPSReports/
report1234.pdf• http://usersUrl/Memos/reminder_03-
2014.txt
http://usersUrl/results.aspx?k=tps
Auto-magically mapped to current zone
User quer
y
Web01 Query Results:
• http://non-default-url/foo/DispForm.aspx?ID=3• http://non-default-url/TPSReports/
report1234.pdf• http://non-default-url/Memos/reminder_03-
2014.txt
http://non-default-url/results.aspx?k=tps
Web App B
Web App A
Query Results:• http://morefoo/foo/DispForm.aspx?ID=3• http://morefoo/TPSReports/report1234.pdf • http://morefoo/Memos/reminder_03-2014.txt
http://spdemo/results.aspx?k=tps
Intranethttp://spdemo
Defaulthttp://sptest
AlternateAccess
MappingsIntranethttp://morefoo
Defaulthttp://foo
Queries across Web Apps use same zone
Demo 1. Test query from Search Center2. Test contextual query (e.g.
“This Site”)
Search Index
Items in Index
Why this does not work…
OriginalPath(string): http://non-default-url/Docs/foo.pptx
Path(string): http://non-default-url/Docs/foo.pptx
SitePath(string): http://non-default-url/Docs/foo.pptx
ParentLink(string): http://non-default-url/Docs/AllItems.aspx
ServerRedirectedUrl(string):http://non-default-url/_layouts/PowerPoint.aspx
SPSiteURL(string):http://non-default-url
QPCQuery Processing
Component
Query comes in via http://non-default-url
Look for content with root
http://usersurl
Demo Break it by manually swapping AAMs
Search Index
Items in Index
Server Name Mappings are not intended for SharePoint Content
OriginalPath(string): http://usersUrl/Docs/foo.pptx
Path(string): http://usersUrl/Docs/foo.pptx
SitePath(string): http://usersUrl/Docs/foo.pptx
ParentLink(string): http://nonDefaultURL/Docs/AllItems.aspx
ServerRedirectedUrl(string):http://nonDefaultURL/_layouts/PowerPoint.aspx
SPSiteURL(string):http://nonDefaultUrl
OriginalPath(string): http://usersUrl/Docs/foo.pptx
Path(string): http://usersUrl/Docs/foo.pptx
SitePath(string): http://usersUrl/Docs/foo.pptx
ParentLink(string): http://non-Default-URL/Docs/AllItems.aspx
ServerRedirectedUrl(string):http://non-Default-URL/_layouts/PowerPoint.aspx
SPSiteURL(string):http://nonDefaultUrl
MPs have inconsistent root URLs when using
Server Name Mappings
Five key takeaways from this session…Application Server Administration Service Timer JobSynchronizes search instances across the farm
Crawl state lives in the databaseMore precise state can be found in the MSSCrawlHistory/MSSCrawlComponentsState tables
Follow a crawled document via ULSUse VerboseEx ULS to track from Crawler to CPC to Index
Follow a query via ULSUse Merge-SPLogfile, Query starts at the Web Front End
Crawl the default zoneDon’t use server name mappings for SharePoint content
See you at the Search booths & Search tables at Ask the Experts WED @6:15pm!
Session Session Room Time
Develop Advanced Search-Driven SharePoint 2013 Apps SPC402 Palazzo I, J Tue 1:45pm
Best practices for Hybrid Search deployments SPC306 Veronese 2401 Tue 5:00pm
SharePoint 2013 Search Analytics SPC340 Palazzo M, N Wed 9:00am
How to manage and troubleshoot Search: A practical guide SPC375 Veronese 2401Wed 10:45am
6 Proven Steps to Get the Best Out of Search in SharePoint 2013 SPC265 Delphino 4001 Wed 1:45pm
Best practices for Information Architecture and Enterprise Search SPC207 Veronese 2401 Wed 1:45pm
Search content enrichment and extensibility in SharePoint 2013 SCP414 Palazzo K, L Wed 1:45pm
Customizing Search experiences with Azure Hosted Data and Bing Maps SPC321 Veronese 2401 Wed 3:15pm
Futuristic Search applications using Kinect and Yammer! SPC405 Palazzo M, N Wed 3:15pm
Search architecture and sizing in SharePoint 2013 SPC336 Titian 2201 Wed 5:00pm
Effective Search deployment and operations in SharePoint 2013 SPC360 Veronese 2401 Thu 9:00am
SharePoint 2013 Search display templates and query rules SPC322 Palazzo M, N Thu 9:00am
Managing Search Relevance in SharePoint 2013 and O365 SPC382 Veronese 2401 Thu 12:00pm
Searc
h R
ela
ted S
ess
ion
s
MySPCSponsored by
connect. reimagine. transform.
Evaluate sessionson MySPC using yourlaptop or mobile device:myspc.sharepointconference.com
© 2014 Microsoft Corporation. All rights reserved. Microsoft, Windows, and other product names are or may be registered trademarks and/or trademarks in the U.S. and/or other countries.The information herein is for informational purposes only and represents the current view of Microsoft Corporation as of the date of this presentation. Because Microsoft must respond to changing market conditions, it should not be interpreted to be a commitment on the part of Microsoft, and Microsoft cannot guarantee the accuracy of any information provided after the date of this presentation. MICROSOFT MAKES NO WARRANTIES, EXPRESS, IMPLIED OR STATUTORY, AS TO THE INFORMATION IN THIS PRESENTATION.
Related Documents
