PowerShell from the Trenches Three Examples of solutions implemented with PowerShell Presented by: Jorge Besada - SQL Server Database Administrator from Carnival Cruise Lines email: [email protected] Other qualifications: See next page for email signature
Welcome message from author
This document is posted to help you gain knowledge. Please leave a comment to let me know what you think about it! Share it to your friends and learn new things together.
Transcript
PowerShell from the Trenches
Three Examples of solutions implemented with PowerShell
Presented by:Jorge Besada - SQL Server Database Administrator from Carnival Cruise Lines email: [email protected] qualifications:See next page for email signature
Jorge Besada - CPE, MPD(Carnival PowerShell Evangelist - Master PowerShell Developer)------------------------------------------------------------------------------From Jeff Snover, PowerShell creator"It's like programming with hand grenades.”
First application
Here at work we make extensive use of SAN storage and many of the server drives are not local drives. They work well, we forget about them until one of them fails. 99% of the time is the problem is not an actual failure, but the connection/mapping of the drive/SAN is lost.
And when the drive that goes down happens to contain the SQL server installation of an instance you get a lot of calls from angry customers and it is difficult to diagnose initially (especially if it has been a long time since the last incident of missing drives) what actually happened. When you miss a drive holding user database files is not that bad, you see the database as offline and eventually you find the cause.The fix is easy: the network person just does a rescan
From command line:echo rescan | diskpart
So to save ourselves DBAs some aggravation a powershell program was created that does the following:It lists all drives from a list of serversIt compares the list of today’s run with the saved list from yesterday’s run, the differences will be the servers with missing drivesIf there is a difference it runs the rescan command on the affected serversAn email is sent reporting the issue
• #GetMissingDrives.ps1• #sends email with servers and missing drives• #rescans drives
• #how to use it:• #Edit the file servers_list.dat to include the list of servers to check
(a server per line)• #(comments can be added starting with #)• #Sample file:
• ##Server list to monitor hard disk space• ##To add comments start the line with #• ##---------SQLSERVERS-----------------• #CCLTSTINFSQL1• #CCLDEVINFSQL1
• $SERVERLIST = 'C:\Users\jorgebe\Documents\powershell\servers_list.dat'• $EMAILTOLIST = '[email protected]'• $FROM = "[email protected]"• $SUBJECT = "Servers Missing Drives Report"• $LOGFILE = 'C:\Users\jorgebe\Documents\powershell\today.lst'• $REFERENCE = 'C:\Users\jorgebe\Documents\powershell\reference.lst'
• function ListDrives($Servername)• {• ForEach-Object `• {• get-wmiobject -computername $Servername win32_logicaldisk | select-object systemname, deviceID, Size, Freespace,
DriveType • }• }
• #Returns array of servers and drives• <# Sample output• SERVER1|C• SERVER1|D• SERVER2|C• SERVER2|D• SERVER2|E• ...• #>• function ProcessServers($Serverlist)• {
• $outarray = @()• $Computers = Get-Content -Path $Serverlist;• foreach ($z in $Computers) • {• if ($z -gt "" -and $z[0] -ne "#")• {• $erroractionpreference = "SilentlyContinue"• $a = ListDrives($z)• foreach ($k in $a)• {• if ($k.DriveType -eq 3)• {• $p = $k.deviceid -replace(":", "")
$outarray += $z + '|' + $p
• }• }• • }• }
return $outarray
• • }
Send mail goodie• #Usage: SendMail message • # emailist (as array)• # c:\attach1.txt;c:\attach2.txt • # [email protected] • # "This is the subject"• function SendMail ($report,$emailarray,$attacharray,$from,$subject)• {• $smtpServer = "smtphost.carnival.com"• $msg = new-object Net.Mail.MailMessage• $smtp = new-object Net.Mail.SmtpClient($smtpServer)• $msg.From = $from • foreach ($c in $emailarray)• { • $msg.To.Add($c)• } • $attlist = $attacharray.Split(";")• foreach ($c in $attlist)• { • $att = new-object Net.Mail.Attachment($c)• $msg.Attachments.Add($att)• } • $msg.Subject = $subject• $msg.Body = $report• $smtp.Send($msg)• }
The program
• Write-Host "Processing server list " $SERVERLIST• #remember: server list has the simple list of servers, no drives• #SERVER1• #SERVER2
• #this call expands the original server list to include the drives• #SERVER1|C SERVER1|D SERVER2|C SERVER2|D SERVER2|E
• $r = ProcessServers $SERVERLIST
• #And we save the actual list to a file today.lst• Set-Content $r -Path $LOGFILE -Force
• #We read the reference list (no missing drives, file reference.lst into a variable• $z = Get-Content $REFERENCE
• #And this is one of the best things of PowerShell: the comparison comandlet• #Compare-Object -ReferenceObject $z -DifferenceObject $r• $g = Compare-Object $z $r
• #if there is a difference that means we lost a drive• if ($g)• {
$msg = [char]13 + 'Re-scanning disks...' + [char]13Write-Host $msg
• foreach ($x in $g)• {• $c = $x.InputObject.Split('|')[0]• Write-Host 'Computer: ' $c• $msg = $msg + $c + [char]13• #And this is another extremely cool powershell thing: The Invoke-Command• #to use invoke-command you need to be at PowerShell version 2.0• $h = invoke-command -computername $c {"rescan" | diskpart}• foreach ($f in $h)• {• $msg = $msg + $f + [char]13• }• }• SendMail $msg $EMAILTOLIST $LOGFILE $FROM $SUBJECT
• #The previous code fixed the missing drives calling rescan for the involved servers• #now we reset the reference list, to leave it ready for the next run• $r = ProcessServers $SERVERLIST• Set-Content $r -Path $REFERENCE -Force• }
Quiz Now
The program as listed in the previous slide has a bug. It works as expected, but the bug is there … lurking. I discovered it while preparing this presentation and I have not yet fixed it
Can you find it?Going back to the code screen …
The little bug: suppose you missed more than one drive in a server. You only need to rescan once for all the missing drives to be reattached. The original code would have done the rescan once for each missing drive.This is fixed in the following code:
• if ($g)• {• $msg = [char]13 + 'Re-scanning disks...' + [char]13• Write-Host $msg• $c_previous = "" • foreach ($x in $g)• {• $c = $x.InputObject.Split('|')[0]#here we get the computer name• #Here is the fix:• if ($c -ne $c_previous)• {• Write-Host 'Computer: ' $c• $msg = $msg + $c + [char]13• $h = invoke-command -computername $c {"rescan" | diskpart}• foreach ($f in $h)• {• $msg = $msg + $f + [char]13• }• $c_previous = $c• }• }
End of First ApplicationApp number two: The Data Quest
High level manager request data to be pulled from all ships. After a lot of work two queries were created. The task was to run these two queries in every ship and collect the results in csv files- The queries may have to be re-run after
updates- Satellite connections to the ships are slow, and
sometimes not available
First try
A cmd file was created calling sqlcmd for each ship, executing the two queries on each ship sql server and saving the output to a file:
One_ship.cmd:Sqlcmd –S %1 –s, -W –d ssadserver –I”cpg_query1.sql” –o%1”_1.csv”Sqlcmd –S %1 –s, -W –d ssadserver –I”cpg_query2.sql” –o%1”_2.csv”
All_ships.cmd:One_ship.cmd SHIP1One_ship.cmd SHIP2… many ships
This did not work: the results coming across the network (some files were 60Mb and more), the process hung for hours.We needed a program to do this:- Copy the queries to the ship- Execute them in the ship, keep results in ship
- Compress the output file- Copy results back to office
- All these steps could have been done with cmd files, but I have noticed that I save a lot of time using powershell combined with cmd files. I have a simple rule for this:
- can the program be done with a single bat file- If yes, just do it - If not, most of the time you save time using
powershell as a driver of other scripts
• #CPG_Transfers2.ps1 FILE SHARE VERSION, Jorge's workstation version• #Runs sql queries in ships, collects resulting csv files
• #How to use it:• #Edit the file servers_list.dat to include the list of servers to check (a server per line)• #(comments can be added starting with #)• #Sample file:
• ##To add comments start the line with #• ##---------SQLSERVERS-----------------• #SHIPSERVER1• #SHIPSERVER2
• $SERVERLIST = 'C:\Users\jorgebe\Documents\powershell\ships_list.dat'• $EMAILTOLIST = '[email protected]'• $FROM = "[email protected]"• $SUBJECT = "CPG Transfer Report - Fileshare version"• $SAVE_FOLDER = '\\CCLDEVINFSQL1\TEMP\backup\'
• #Returns ping number or empty string if no response• function CheckPing($server)• {
$v = (ping $server -n 1)
• foreach ($k in $v)• {• if ($k.StartsWith("Reply"))• {• break• }• }• $l = $k.Replace('<', '=')• $lst = $l.split('=')[2]• if ($lst)• {• $r = $lst.Replace('ms TTL', '')• }• else• {• $r = ""• }• return $r• }
• function ProcessServers($Serverlist)• {
• $outdict = @{}• $Computers = Get-Content -Path $Serverlist;• foreach ($z in $Computers) • {• #Sort by ping process• if ($z -gt "" -and $z[0] -ne "#")• {• $ping = CheckPing($z)• $outdict[$z] = [int]$ping• }
• }
• $sorted = $outdict.GetEnumerator() | Sort-Object Value
• foreach ($j in $sorted)• {• $t1 = Get-Date• $dt1 = $t1.tostring()• Write-Host "Started" $dt1
• $ping = $j.value• $z = $j.name• $ping_current = CheckPing($z)• Write-Host "Ping response original run: " $ping" Current: " $ping_current• if ($ping_current -gt "")• {• Write-Host "Copying queries to ship" $z• $dest = '\\' + $z + '\DBAScripts'• copy "cpg_query1.sql" $dest -Force• copy "cpg_query2.sql" $dest -Force
• Write-Host "Executing CPG queries on ship"• (C:\Users\jorgebe\Documents\powershell\cpg_transfers.cmd $z)
• Write-Host "Compressing results"• (C:\Users\jorgebe\Documents\powershell\cpg_transfers_compress.cmd $z)
• Write-Host "Collecting results"• $src1 = $dest + '\cpg_query1.csv'• $src2 = $dest + '\cpg_query2.csv'• copy $src1 ($SAVE_FOLDER + $z + '_1.csv') -Force• copy $src2 ($SAVE_FOLDER + $z + '_2.csv') -Force
• $SUBJECT = $z + " cpg queries process completed"• $msg = "Started at " + $dt1 + [char]13 + "Completed at " + $dt2 + [char]13• $tt = $t2 - $t1• $msg = $msg + "Elapsed time: " + $tt.ToString()+ [char]13• $msg = $msg + "Ping response: " + $ping + [char]13• SendMail $msg $EMAILTOLIST "" $FROM $SUBJECT• }• else• {• Write-Host "-------------------------------------"• Write-Host "Ship " $z " not responding"• Write-Host "-------------------------------------"• }
}
Vodoo cmd file (cpg_transfers.cmd)The double combination of the 2 sqlcmd calls and xp_cmdshell enables the placing of the output file in the desired remote server location
Line 1:
sqlcmd -S %1,9999 -l 10000 -Q“xp_cmdshell ‘sqlcmd -h -1 -l 10000 -S . -d ssadserver -s, -W -I -id:\DBAScripts\cpg_query1.sql -o d:\DBAScripts\cpg_query1.csv‘“
Line 2:
sqlcmd -S %1,9999 -l 10000 -Q“xp_cmdshell ‘sqlcmd -h -1 -l 10000 -S . -d ssadserver -s, -W -I -id:\DBAScripts\cpg_query2.sql -o d:\DBAScripts\cpg_query2.csv‘“
Remote compress cmd file cpg_transfers_compress.cmd:“compact” is a little known compress utility, it is built in latest server versions
sqlcmd -S %1,9999-l 10000-Q"xp_cmdshell 'compact /c d:\DBAScripts\cpg_query1.csv'“
sqlcmd -S %1,9999-l 10000-Q"xp_cmdshell 'compact /c d:\DBAScripts\cpg_query2.csv'"
SAMPLE OUTPUT FOR FOUR SHIPS
Thursday, March 14, 2013 3:04:22 PMProcessing ships list ships_list.datPing response: 695 Ship: LESQL1.LEGEND.CARNIVAL.COMCopying queries to ship3/14/2013 3:04:23 PM3/14/2013 3:05:04 PMExecuting CPG queries on ship3/14/2013 3:07:00 PMCompressing results3/14/2013 3:07:19 PMCollecting results3/14/2013 3:08:13 PMcompleted 3/14/2013 3:08:13 PMPing response: 840 Ship: SPSQL1.SPIRIT.CARNIVAL.COMCopying queries to ship3/14/2013 3:08:15 PM3/14/2013 3:08:38 PMExecuting CPG queries on ship3/14/2013 3:09:42 PMCompressing results3/14/2013 3:09:58 PMCollecting results3/14/2013 3:10:39 PMcompleted 3/14/2013 3:10:39 PMProcess completed Thursday, March 14, 2013 3:10:40 PM
Thursday, March 14, 2013 4:18:01 PMProcessing ships list ships_list.datPing response: 1517 Ship: IMSQL1.IMAGINATION.CARNIVAL.COMCopying queries to ship3/14/2013 4:18:03 PM3/14/2013 4:18:55 PMExecuting CPG queries on ship3/14/2013 4:25:47 PMCompressing results3/14/2013 4:26:03 PMCollecting results3/14/2013 4:43:25 PMcompleted 3/14/2013 4:43:25 PMPing response: 799 Ship: SESQL1.SENSATION.CARNIVAL.COMCopying queries to ship3/14/2013 4:43:27 PM3/14/2013 4:43:55 PMExecuting CPG queries on ship3/14/2013 4:49:34 PMCompressing results3/14/2013 4:49:50 PMCollecting results3/14/2013 5:55:56 PMcompleted 3/14/2013 5:55:56 PMProcess completed Thursday, March 14, 2013 5:55:56 PM
App Number Three:Original Request
• There was an old Vb.net application used to send files from corporate office to the remote locations and collect other files from them. It used MSMQ. At some time the application started to have intermittent failures and after some research it was detected that the files had grown overtime to sizes that exceeded the message queue limits
Original Plan “A”Solution
• Add code to the old VB.net application to incorporate file splitting before the transmission and file joining after reception
• A couple of PowerShell functions were created to do the split/join
SplitDeveloped using advanced techniques (next slide)
function split($inFile, $outfolder, $outPrefix){
$bufSize = (Get-ItemProperty $infile).Length / $CHUNKSWrite-Host 'bufsize = ' $bufSize
$stream = [System.IO.File]::OpenRead($inFile) $chunkNum = 1 [Int64]$curOffset = 0 $barr = New-Object byte[] $bufSize
while( $bytesRead = $stream.Read($barr,0,$bufsize)){ $outFile = $outfolder + '\' + "$outPrefix$chunkNum" $ostream = [System.IO.File]::OpenWrite($outFile) $ostream.Write($barr,0,$bytesRead); $ostream.close(); echo "wrote $outFile" $chunkNum += 1 $curOffset += $bytesRead $stream.seek($curOffset,0); }}
rm ($WORKFOLDER + '\*') -Force split $SOURCEFILE $WORKFOLDER $FILEPREFIX
Advanced Technique
• I learned this technique in TechEd 2012 from the PowerShell gurus
Steal from The BestAndCreate the Rest
Join function join($outfile, $folder, $prefix){$cnt = (Get-ChildItem $folder).Count$ostream = [System.IO.File]::OpenWrite($outFile)$i = 1while ($i -lt $cnt + 1){Write-Host $i$file = $folder + '\' + $prefix + $i$bufSize = (Get-ItemProperty $file).LengthWrite-Host $file $bufSize$stream = [System.IO.File]::OpenRead($file) $barr = New-Object byte[] $bufSize
while( $bytesRead = $stream.Read($barr,0,$bufsize)){ $ostream.Write($barr,0,$bytesRead);}$i++}$ostream.close();}
Intermezzo
• All this is fine and dandy, but …• This thing of taking an old vb.net app, adding a
powershell hack to it looks like a … hack
• Can we have a more elegant approach using PowerShell for everything?
• Let’s see the “Plan B”
FileExchanger.ps1- What it does:
The contents of the source main folder \\MAINSERVER\outbound_miamiare sent to the shared folders in the remote servers, each server receives the corresponding filesbased on the file id in the file name to be sentThe contents of the source main folder \\MAINSERVER\inbound_miamireceives the files from all the remote servers
Remote servers:The contents of shared outbound folder (sample folder \\REMOTESERVER\outbound )are sent to the mainserver collecting folder \\MAINSERVER\inbound_miami
How to test:
First:Place files with the correct file name format in the remote server folderSample values:For remote server xxdevsql1 file id used was 300Sample file name is 000000_300_MOP_15269.zip in \\xxdevsql1\app\outboundFor remote server ccltstsql1 file id is 350
Second:Place files with the correct file name format in the mainserver outbound folderSample values:300_000000_MOF_12010.zip350_000000_MOF_12012.zip375_000000_MOF_11256.zip400_000000_CORP_9897.zip
Third: Execute the powershell program FileExchanger.ps1 from MAINSERVER
Results:The files from remote servers will end up in the mainserver inbound folderThe files from the mainserver outbound folder will end up in the corresponding inbound folder in the remote serversThe extra files in the mainserver outbound folder (for remote servers not in the list) are not processed.
FileExchanger.ps1 – the code - 1
#MAINSERVER SHARES$FiletoSendLoc1 = '\\ccldevsql1\outbound_miami'$PlacetoSave1 = '\\ccldevsql1\inbound_miami‘
$REMOTESERVERS = @{300="Miami|xxdevsql1"325="New York|ccltstsql1"350="Rio de Janeiro|ccltstcorpsql1"400="London|ccldevinfsql1"}
function CheckPing($server){$v = (ping $server -n 1)$l = $v[2].Replace('<', '=')$lst = $l.split('=')[2]$r = $lst.Replace('ms TTL', '')return $r}
function ExtractItems($file_id){$server_code, $server_name = $REMOTESERVERS[$file_id].Split('|')return $server_code, $server_name}
FileExchanger.ps1 – the code - 2
function SendFiles($file_id){ Write-Host 'Sending files to ' $REMOTESERVERS[$file_id] $server_code, $server_name = ExtractItems($file_id) Write-Host 'Server name: ' $server_name ', Server code: ' $server_code $dest = '\\' + $server_name + '\app\inbound' $p = $FiletoSendLoc1 + '\' + $file_id + '_*.zip' Write-Host 'destination folder: ' $dest copy $p $dest -Force -Verbose}
function ReceiveFiles($file_id){Write-Host 'Receive files from ' $REMOTESERVERS[$file_id]$server_code, $server_name = ExtractItems($file_id)Write-Host 'Server name: ' $server_name ',Server code: ' $server_code $src = '\\' + $server_name + '\app\outbound\000000_' + $file_id + '_' + '*.zip'Write-Host 'destination folder: ' $PlacetoSave1copy $src $PlacetoSave1 -Force -Verbose}
function MainLoop{foreach ($file_id in $REMOTESERVERS.get_keys()){Write-Host 'Processing' $file_id$server_code, $server_name = ExtractItems($file_id)$t = CheckPing($server_name)Write-Host 'Checking ping response for the record:' $server_name $t 'ms'if ($t -lt 1000){SendFiles($file_id)ReceiveFiles($file_id)}else{Write-Host 'Ping response too high, skipping ' $server_name ' in this run' }}
}
MainLoop
FileExchanger.ps1 – the output
Final Notes
I was fortunate to attend TechEd 2012 this year in Orlando.
It was a highly technical event, I attended most of the PowerShell conferences:
Final Notes
Ooops … wrong slide here
Jeff Snover – PowerShell Creator
... Snover, (the architect behind the product now responsible for the entire windows server platform):
"It's like programming with hand grenades.”
I know people!
Conclusion
If you want to be a good PowerSheller …
You Better Wear Vibram Shoes
Related Documents
