SharePoint Speedmetal for Admins by Chris McNulty - SPTechCon

SharePoint SpeedMetal – Admin 101

SPTechCon San Francisco 2013Chris McNulty

25 yearsSince 1987, Quest has offered a broad and deep selection of products that target common IT challenges

3,900Quest employees develop solutions in over sixty offices throughout the world

18%Quest has driven innovation by regularly investing 18% or more in R&D

30Over thirty acquisitions have strengthened our product portfolio

100kQuest products provide over 100,000 customers with IT solutions every day

3 Confidential SharePoint

Dell Software | SharePoint

Chris McNulty • SharePoint BU at Dell Software

• 10+ years with SharePoint

• 20 years consulting (led KMA SharePoint practice) and financial services technology (Santander, John Hancock/Manulife, GMO, State Street)

• MBA in Inv Mgmt from Boston College

• Write and speak often on Microsoft IW technologies (blogs & books)

• MCSE MCTS MSA MVTSP MCC

• Hiking, cooking, playing guitar, colonial history, photography

• My family: Hayley, three kids (18, 9, 6) and my dog Stan

Hometown, Laurel, NY (from www.flickr.com/photos/cmcnulty)

8 Confidential SharePoint BU

Key Topics

The dilemma

SupportMonitoringOptimizationBackupPowerShellDevelopment PatchingSQL Maintenance

Architecture, design and planning

Installation and upgrade

Best/Worst Practices

Presentation Governance

• In scope– 2010 Administration “Core”

• Out Of Scope– 2013 Deep Dives– Deep Dives (e.g. PowerShell, BI, Upgrade, SQL DBA)– Development/Customization– Power User (e.g. Library Customization, Designer Workflows, etc.)

• Rules– Move fast, PowerPoint is shared – http://slidesha.re/xTcZYq – Questions – time permitting during session– Any time after session – email etc. - @cmcnulty2000

Congratulations!

• You’re the new SharePoint Administrator!!!

• But…

• You’re still responsible for:– Exchange– Active Directory– SQL– Desktop– Help Desk– Network/Firewall– Cooking & Cleaning– Etc.

The Dilemma

• SharePoint administration is often an ‘add-on’ for other IT professionals (SQL DBAs, AD Admins, Exchange Engineers)

• Time and focus are scarce resources!

• Common pain points include– Upgrades are complex and hard to monitor– Dispersed workforce, little control of browsers and Office versions– Hard to understand and troubleshoot “behind the scenes” performance and

capacity planning– Best practices not always understood or compared to system health– “All or nothing” administration means IT must be engaged for all admin

responsibilities, even search

Microsoft SharePoint Server 2010 … the bright frontier

Eastern Long Island, July 4, 2010

Architecture and Design

Server Farm – Web Front End

• Typical Roles:– http services– Search query

• Scaling– Add servers to load balanced

cluster

• Performance Optimization– RAM– Easily virtualized

Server Farm - Application Server

• Typical Roles:– Search index/crawl– Excel calculation– User profiles– Managed Metadata

• Scaling– Add search servers and partitions– Move shared services to

dedicated servers

• Performance Optimization– CPU

Server Farm - Database

• Typical Roles:– Data storage– SQL Reporting

• Scaling– Add storage capacity

• Performance Optimization– Disk I/O

Sizing - Single Server

• Typical Roles:– Small teams– Small pools of documents

• Considerations– Performance & fault tolerance less of a

concern– SQL & Web on same system– Search not a core function

Sizing - Medium Farm

• Typical Roles:– 100-10,000 users– 10,000 – 1MM documents

• Scenarios– Enterprise portal– Large scale collaboration– Broader applications platform– Larger external search pool– Mix and match internal external front end

servers on common content databases

Sizing - Large Farm

• Typical Roles:– Large distributed enterprise

users (10000+)– Large pools of documents

(>1MM)

Sizing – No Servers – Office 365

Office 365 Enterprise Plans

E1 E2 E3 E4

SharePoint Online √ √ √ √

Office Web Apps √ √ √

Local Copy of Office Professional 2010 Plus

√ √

Forms Services, Vision Services, Access Services

√ √

Monthly cost per user $10 $16 $24 $27

• Constraints/Unavailable– Custom, non sandbox solutions– Power Pivot– SQL Server Reporting Service

Integration– Business Connectivity Services

(OK for web services- based remote data in O365 BCS.)

– FAST Search Server Integration– Web Analytics– Site collections greater than

Web Applications

http://

intrane

Site Collections

Site Hierarchies

• Farm level

• Web applications – Independent top level URLs– Run inside IIS pools– Consume shared services and

admin from the farm or other farms

• Site collections – Security, branding, database

frontier– Contain single sites or site

hierarchies

• Sites– Group related SharePoint

elements (lists, libraries, pages, web parts)

Top Level Logical Components

• High capacity!

• Maximums– 250,000 sites per site collection– 5,000 site collections per content DB– 200GB max content DB (single site

collection)– >200GB post SP1– 300 Content DBs per web application– 30MM documents/library– 2GB document size

• 2011 News– 14TB Demo

• 2013 Notes– Shredded storage?

Logical Components

Disk Sizing

Content Search

Initial Content Size XXX GB External Crawl Size YYY GB

Initial User Pool U

User Collab Size .25GB

n YR Growth Rate – Archive Rate G%

End Content Size XXX (1+G)n = ECS End Search Size YYY (1+G)n = ESS

End User Collab Size .25 * U * (1+G)n = EUCS

Content DBs ECS + EUCS

Search DBs .05 * (ECS + EUCS + ESS)

Search Index Files .05 * (ECS + EUCS + ESS)

• Inputs: Size of SharePoint content and non-SharePoint content included in search• For DBs, don’t forget transaction logs, disk dumps (if used for backup) which can add 1-3X.• In SAN or virtual environments, not all disk need be provisioned early

Content will grow!Searching remote stores saves disk but isn’t free.

Database sizing

• Automatic database growth:– 50-100MB clumps – not by percentage. – Using a small size leads to more frequent, but smoother, steady state

growth.

• Presize tempdb to about 20% the size of the single largest content database.

Psst…about SQL…

• Sidebar on SQL Disk optimization– RAID 1: Mirroring (Wastes disk)– RAID 5: Parity Bit (write performance hit)– RAID 10 Stripes across mirrors (costly)

• Physical location of data, log, temp and/or backup files. If virtualization or SAN technology doesn’t isolate the disks, not much performance gain,

• Performance optimization/fault tolerance by:– RAID1 on boot disks– RAID5 on data disks– RAID10 on log disks– No RAID, or RAID 5 on backup disks

• RBS reduces size (and count) of content databases but doesn’t reduce size of total storage

• IOPS, IOPS, IOPS!

Internet Topology – Edge Firewall

• Traditional

• Inexpensive

• Simple

• Only one firewall

• External traffic comes inside internal network

WFE SQLEdge

Firewall AD

Internet

Internal Network

Internet Topology – Perimeter

• More complex

• Duplicative networks, backup, AD

• External traffic is reserved

• Larger server foot print (exposure) in perimeter

• Internal users need domain trusts

• Internal users access site across firewall

Router/Firewall

WFE SQLEdge

Firewall AD

Internet

Perimeter Network Internal Net

Internet Topology – Split Back to Back

• Most complex

• Intricate firewall rules

• App, AD and search roles optionally in perimeter

• Optional internal WFE or internal users always cross a firewall

• Crawl topologies important to avoid overtaxing the firewall

Router/FirewallWFE SQL

EdgeFirewall AD

Internet

Perimeter Network Internal Network

Internet Topology – Enhanced Techniques

• Multi-farm

• SSA farm

• Content publishing

Platform Basics

• SharePoint 2010 is a 64 bit only platform.

• Direct upgrades from 32 bit to 64 bit require prep work.

• Windows Server 2008 or Windows Server 2008 R2 X64– SQL Server 2005 x64 SP3 CU3

Or– SQL Server 2008 x64 SP1 CU2

Or– SQL Server 2008 R2

Shared Service Applications

• 2007 Shared Services Provider has been broken up; each of its elements is now a Shared Service Application

• Mix and match them singly or in groups, to match farm’s needs.

• Crawl/index no longer a single server role

• In 2010, administration can be delegated– Key targets: Enterprise

search, metadata, user profiles

http://globalweb http://itportal

Search

Excel Calc

Metadata

User Profiles

Client/Browser Technology

• Internet Explorer 7/8/9, Firefox and Safari are all supported.

• Some support for Chrome

• IE6 is not supported

• Most other browsers are still supported for Internet configurations

• Office 2010 includes optimizations for the new platforms

• Offline Access– 2007: used Outlook 2007 and Groove– SharePoint Workspace 2010 integrates offline documents and lists

Office Web Applications

• SharePoint 2010 provides a server version of Office applications – Office Web Access, or “OWA”.

• In part, this enables simultaneous multiuser editing of Office documents:

– Excel in OWA, not client– Word/PowerPoint on client only if file

opened from a shared document library– OneNote client or OWA

Installation and Upgrade

Installation - Prerequisites

• Servers:– Windows 2008 R2 X64 Enterprise Edition– SQL Server 2008 R2 x64

• Service Accounts– spfarm (Farm acct; local admin on the SharePoint servers and either sa or dbcreate, dbowner and

security admin on the SQL server.)– svcsql (SQL Server service acct)– sppool (IIS pool acct)– spcrawl (Search accts)– spadmin Interactive admin (install account; local, site collection and farm admin privileges)

• Install as SPAdmin

• Install Software Prerequisites - Checks for following elements:– Application Server Role, Web Server (IIS) Role, Microsoft SQL Server 2008 Native Client, Hotfix for

Microsoft Windows (KB976462), Windows Identity Foundation (KB974405), Microsoft Sync Framework Runtime v1.0 (x64), Microsoft Chart Controls for Microsoft .NET Framework 3.5, Microsoft Filter Pack 2.0, Microsoft SQL Server 2008 Analysis Services ADOMD.NET, Microsoft Server Speech Platform Runtime (x64), Microsoft Server Speech Recognition Language - TELE(en-US), SQL 2008 R2 Reporting Services SharePoint 2010 Add-in

Installation – Grey Wizard

• Initial– Product Key– Type of installation - Always SERVER FARM– Installation Type - Complete [Not Single

Server]– Accept default file locations – index files

will stay on C:\Program Files\Microsoft Office Servers\14.0\Data

– At end NO Wizard– Run OWA Setup– Then, WIZARD! The wizard starts, and

yes, it’s OK for IIS to reset during the wizard…

• Create a new farm– Set farm account– Pick configuration database, Passphrase,

CentralAdmin Port (Conventions)– Final confirm and let the wizard run

Installation – White Wizard?

• Pros– Easy – shaken and stirred– All SSAs Configured– Saves time and PowerShell hand

tooling of SSAs

• Cons– My Sites setup in same app and

DB as primary– Database Names are default,

GUID happy

• What it does– Sets up service acct for SSAs and

other services (sppool)– Sets up a port 80 web app with a

My Sites Host sub-site collection in WSS_Content database

Predictable Upgrade

• Three paths– In place– Database upgrade– Third party tools

• Process– Pre-upgrade checker– Visual Upgrade– Resumable upgrade– Progress reports– Parallel DB upgrades

Upgrade Preparation

• Additional Prepwork– Content pruning

– Database alignment

– stsadm-o mergecontentdbs

• DB Attach

– Preinstall Required Features

– Stsadm –o addcontentdb –databasename DBNAME –url URL –assignnewdatabaseid

– PowerShell Mount-ContentDatabase

– Test, test, test!

Pre-Upgrade Check

• SharePoint 2007 SP2 minimum, October 2009 CU best

– STSADM.exe –o preupgradecheck

• Documentation– All servers and components in the farm, and whether

the servers meet 64-bit hardware/OS requirements– Alternate access mapping URLs– Site definitions, site templates, features, and language

packs– Unsupported farm customizations (such as database

schema modifications).– Database or site orphans – Missing or invalid configuration settings in the farm

(missing Web.config file, invalid host names, invalid service accts).

– Whether the databases meet the requirements — for example, databases are set to read/write, and any databases stored in Windows Internal Database and larger than 4 GB.

• Doesn’t exist for 2010 – 2013!

Upgrade servers without changing the

user interface

Switch-on new UI across site collections

in a controlled manner

Preview new UI

IT Pro Investments – Visual Upgrade

SharePoint

DemoUpgrade

Monitoring and Operations

Proactive Issue Resolution

• Developer Dashboard– Empower developers and users

• Integrated Health Analyzer– Runs when necessary– Alerts anomalies – Fixes when it can

• Web Analytics– User usage– Resource usage

Logging, Monitoring, and Alerts

• Unified Logging

• Out-of-the-box reports

• Richer Web Analytics

• Open Schema

• SCOM Integration

• PLUS– Developer Dashboard– Health Analyzer

ULS Logs

Windows Events

Page requests

Feature Logging

Health data

Logging DB

Developer Dashboard

Improve customized solutions with the Developer Dashboard

SQL Queries Performance

Memory Usage

Data-Request Trace

Whoops!

• Default user screen

• Correlation ID is shown!

Monitoring – WSS_Logging

• Query Database Views Directly

• Requires Timer Jobs Enabled– Diagnostic Data Provider: Trace Log– Diagnostic Data Provider: Event Log

• ULS Configuration Matters

• Database will GROW!

• Aggregates from ALL Servers

• Sample: – SELECT * FROM [WSS_Logging].[dbo].

[ULSTraceLog] WHERE CorrelationID = '04377DAE-C2FD-4DBE-A57E-101B3005059E'

Development Support – Three Regions

Development

•often internal to developers•problem reproduction that require advanced inspection tools (e.g. Visual Studio) are done here•permissions can be looser, may have multiple environments for multiple developers•sensitive data from production cannot be copied here without masking or customer signoff•changes here can be deployed ad hoc

Staging/Test

•no Visual Studio, no MS Office•match/mirror production as closely as possible; match hardware/system performance as closely as practical

•security permissions match production

•any sensitive data copied here stays under production-grade controls

•test accounts should be created in a separate OU if possible

•changes here can only be delivered and deployed from source control and according to production release methods

Production•optimized hardware configurations•highly secure•no use of user rotating password accounts as service accounts•changes here can only be delivered and deployed from source control and according to production release methods

Backup/Recovery

• Third Party Tools

• Recycle Bin

• Granular / Site Collection Backup (UI)– *.bak file– Restore-SPSite

• Unattached Recovery– Browse unattached content database– Account needs DB permissions– Database need not be on the same server!– No more granular than list or library!– Browse Content

– Export Site or List

– Export as a CMP file– PowerShell restore

– PS: Import-SPWeb http://msshome2010 –Path C:\ListRecovery.cmp

• SQL Backup

• SharePoint Backup (UI or script)

STSADM Backup

@echo off

echo ==================================================

echo Backup the farm

echo ==================================================

@SET stsadm="C:\Program Files\Common Files\Microsoft Shared\Web Server Extensions\12\BIN\stsadm"

rmdir /S /Q "\\spsql08\spbackup\farmold"

ren "\\spsql08\spbackup\farm" "farmold"

md "\\spsql08\spbackup\farm"

%stsadm% -o backup -directory "\\spsql08\spbackup\farm" -backupmethod full

echo complete

PowerShell Backup

# NOT NEEDED write-output

write-host ==================================================

write-host Backup the farm

write-host ==================================================

Add-PSSnapIn Microsoft.SharePoint.Powershell

Remove-Item -Path "C:\PSBackup\farmold“ -recurse

Rename-Item -Path "C:\PSBackup\farm" -NewName "farmold"

New-Item -type directory -path C:\PSBackup\farm

Backup-SPFarm -directory "C:\PSBackup\farm" -backupmethod full –verbose –percentage 5

Write-host Backup complete

SharePoint

DemoMonitoringDeveloper DashboardHealth AnalyzerLogging and

Reporting

Optimization

• Disk-based BLOB Caching– Local store for audio/video, PDF other frequent read only files– Edit in Web.config (C:\Inetpub\wwwroot\wss\virtualdirectories\...)– <BlobCache location="" path="\.(gif|jpg|jpeg|jpe|jfif|bmp|dib|tif|tiff|ico|png|

wdp|hdp|css|js|asf|avi|flv|m4v|mov|mp3|mp4|mpeg|mpg|rm|rmvb|wma|wmv)$" maxSize="10" enabled="false" />– Location = Local Disk Location– maxSize = GB– Enabled = true

• Different from RBS/EBS!

• For publishing sites

Patching – Process through August 2011

Patch •SharePoint Foundation

Patch •SharePoint Server

Deploy •Run SharePoint Products and Technologies Wizard•(Or psconfig) •Sequential Application to Central Admin, Application Server(s), Web Front End Servers

Patching – Process After August 2011

Patch •Separate patch longer needed – single patch CU now available

Patch •SharePoint Foundation•OR SPF/Server•OR SPF/Server/Project Server

Deploy •Run SharePoint Products and Technologies Wizard•(Or psconfig) •Sequential Application to Central Admin, Application Server(s), Web Front End Servers

Patching – Notes

• Test before installation!!!

• Cumulative Updates every two months

• Service Pack every 6-18 months

• Service Pack 1 REQUIRES at least the June 2011 CU

• December 2011 CU – iOS 5 Mobile Safari support for Performance Point– Fixed Administrator updates to user profile pictures

• February 2013 CU most current

• Check my blog for latest: http://www.chrismcnulty.net/blog/Lists/Categories/Category.aspx?CategoryId=5&Name=Version-Build Numbers

SQL Maintenance

• Backups– Local Disk – easy but storage intensive– Agents – remote, requires extra software

• RBS Maintenance– BLOB Orphans

• Log Sizing– Full logged (default) generates huge t-logs– Simple doesn’t but prevents point in time restore

• Maintenance Plans

• On SharePoint– New-SPWOPIBinding -ServerName

<WacServerName> -AllowHTTP

• Set zone to regular http– Set-SPWopiZone –zone “internal-

http”

• Troubleshooting:– No system account usage!– Configure system to allow OAuth

over http– $config = (Get-

SPSecurityTokenServiceConfig)– $config.AllowOAuthOverHttp =

$true– $config.Update()

• Off-server installation

• Prerequisites– .NET Framework 4.5/ASP.NET 4.5– Ink and Handwriting– Windows Authentication, NET Extensibility

4.5– ISAPI Extension, ISAPI Filters– Server Side Includes

• Default installation (DNS, https)

• PowerShell– Import-Module OfficeWebApps– New-OfficeWebAppsFarm –InternalURL

http://SP2013Demo-WAC –ExternalURL http:// SP2013Demo-WAC.spdemo.corp –AllowHttp –EditingEnabled –ClipArtEnabled

• Test - http://servername/hosting/discovery

Office Web Apps Server

Social Administration - Quotas

• Check quotas based on errors, usage patterns

• Default is 100MB

Best Practices

Troubleshooting – Top Support Questions

• Users Receive “Cannot Connect to Configuration Database” Web Page– SharePoint farm account is locked out

• No one can upload anything but site is up– Database disk volume is full – check transaction logs, backups– In virtualized environment, host file systems may be full

• I can’t find a document I think I should see; Someone can’t see a file I just uploaded– Security and permission variations– Document “movement” (a/k/a ECM) try search by name or Document ID.

Check ECM logs/audits– Confirm permissions, and make sure document is checked in (Required

properties may be missing)

Troubleshooting – More Support Questions

• Repeated requests to re-enter Windows credentials– Add to Local intranet zone, add site, custom level, automatic login with

current user name and password (it’s the last thing in the item list)– OR Trusted sites

• My workflow didn’t start– Recycle timer service– “FixSharePoint.exe” = IISReset & Timer Service Recycle

• I’m not seeing the right search results– Confirm that crawls are running and complete by checking crawl logs;

restart a full crawl if crawls finish OK

• I need a file back that I deleted– Recycle Bin Recovery– Use Backup & Restore

Seven Deadly Sins

• No SQL maintenance plans

• Default names for every database (WSS_CONTENT_12345abc…)

• No patching

• One environment for everything

• One acct for everything

• Single server install with SQL Express

• Runaway content database size

Seven SharePoint Virtues

• Security Applied via AD Groups and SharePoint Groups

• Review System Health

• Test Restore and Recovery

• Monthly Web Analytics Review – Usage, Storage, Search

• PowerShell instead of STSADM

• Governance

• Use ECM, MMS, Clients, Archiving and Training to Keep Content in SharePoint, reduce accidental duplication and keep searching and browsing fresh

Congratulations!

• You’re the new SharePoint Administrator!!!

• And now

• You understand:– Design and Architecture– Installation and Upgrade– Support and Maintenance and

Optimization– PowerShell– Customizations– Troubleshooting– Best Practices

Thank you!

– Questions– Contact

– Email chris_mcnulty@dell.com – Blog http://www.chrismcnulty.net/blog

– Also http://www.sharepointforall.com – Twitter: @cmcnulty2000

– Upcoming:– ICC Heartland Conf, SP Evolutions,

DCSPUG, SPUGME, ShareFest Denver

While you're in the Bay Area…

• Monday 4:00pm - SharePoint SpeedMetal Admin 101

• Monday 5:30pm – Lightning Talks

• Tuesday 2:00pm – Social 101 and SharePoint

• Tuesday 6:00pm – Book Signing, Dell, SharePoint 2013 Consultant’s Handbook (Advance Edition)

• Wednesday 1:15pm – SharePoint Experts Meetup (Managed Metadata)

• Wednesday 3:45pm – I Have Excel, I Need PerformancePoint, but I Don’t Know Analysis Services!

More information

• SharePoint architecture design patterns in Chris’ e-book entitled SharePoint 2010 Consultant’s Handbook – A Practical Field Guide

– Get your free copy here http://www.quest.com/get-chris-book

SharePoint Speedmetal for Admins by Chris McNulty - SPTechCon

Documents

SPTechCon SharePoint Worst Practices

SPTechCon - San Francisco - Is Your SharePoint Healthy?

SPTechCon SFO 2013 - SharePoint Solutions with SPServices

What’s your Social IQ? Succeeding with SharePoint Social.....

SPTechCon - July 2012 - Effective requirements gathering...

Tj mcnulty

SPTechCon Boston 2015 - Overcoming SharePoint Limitations

Paul McNulty PhD Thesis

frances role mcnulty

A Guide for CorpPass Admins & Sub-Admins Agency Training...

SPTechCon BOS 2013 - SharePoint Solutions with SPServices

SPTechCon Boston 2012 - SharePoint Integration and...

Microsoft Keynote by Richard Riley - SPTechCon

SPTechCon Austin 2015 - Perfecting Information Architecture

IPv6 sys-admins

SPTechCon Extending ECM for Developers