BENE Jo Vanvoorden Upgrade je Blackboard 7.2 servers naar 9.0 Implicaties van de upgrade voor systeembeheerders.
Jun 14, 2015
BENE Jo Vanvoorden
Upgrade je Blackboard 7.2
servers naar 9.0
Implicaties van de upgrade
voor systeembeheerders.
Overzicht
• Upgrade pad• Voorbereiding• Servers• Timeframe systeem beheerders• Doing the job• Performance + bevindingen• Nazorg
Upgrade pad
• 7.2.383.0 => 7.2.383.45• 7.2.383.45 => 8.0.260.7• 8.0.260.7 => Xythos Unique ID tool• 8.0.260.7 => 9.0.351.4• 9.0.351.4 => 9.0.505.0• 9.0.505.0 => 9.0.505.0 + Patchset + MR
Voorbereiding
• Gestart met testfase upgrade 8• Pad verder uitgeschreven voor BB9• Verschillende versies getest• Clean install• Full install• 2 FTE +2maanden fulltime restore + testen• Uitbouw hulpmiddelen
– Patch deploy script– Smsmewakker script– Comparesystems script– Installblackboardfromscratch script
Servers
• Applicatie Servers ?
Cygnus (albireo, sadir,gienah,azelfafage,ruchba)
Lyra ( tolbb9app2t)
tolbb9dev1t
tolbb9dev2t
tolbb9dev3t
tolbb9dev4t
tolbb9dev5t
tolbb9dev6t
Muliphein (tolbb9app3t)Tolbb9app1t
Tolbb9app4tAlshain,altair
Canis-major (sirius,mirzam)
Servers
• Applicatie Servers ?• Database Servers
– Tolora10db6, elune, polaris– Tolora10db1t, tolora10db2t, tolora10db3t,tolora10db4t,
tolora10db5t, altair
• Storage– Backend Netapp -> full system restores– DB -> full restores 5 test db servers (500G/server)
• Hardware– Full restores => fysieke hardware– Development restores => vmware esx
Timeframe systeem beheerders
• Testen :– Run1
– Run 2 : 14 h 30– Run 3 : 15 h– Run …13h
upgrade 7.2 -> 7.2.383.45
18min?
upgrade 7.2 -> 8
40min
UNIQUE ID TOOL
10.34 -> 16h gestopt opnieuw gestart in screen
16h -> 21h.... gestopt met errors
2010-01-20T20:53:44Z] Trace XythosSession ContextImpl:201 Transaction Aborted com.xythos.storageServer.api.StorageServerException: ContextImpl:199 Transaction Aborted
(com.xythos.storageServer.api.EntryReadException: FileSystemEntryException on entry bbcms//users/j-1013000000027004/calculations.doc
upgrade 8 -> 9.0.351.4
7h
upgrade 9.0.351.4 -> 9.0.505.0
1.5h
Doing the job
• Vrijdagavond 12 februari - 20.00– Cold backup db + app + snapshots backend, gdlo down +
redirect in place, start restore reference system
• 21.07 DB in noarchivelog + andere voorbereidingen• 21.21 PONR
– 7.2.383.0 => 7.2.383.45
• 21.56– Reconfigure bb-tasks, disable PA ,next installer ( => 8.0.184.3 )
• 22.29– Reconfigure bb-tasks, Unique ID tool
Doing the job
• Zaterdagmorgen• 04.40
– Xythosid finished -> patchen ongedaan maken
• 5.08– java upgrade, classpath aanpassen, blackboard aanpassen +
volgende installer (9.0.351.4)
• 6.33– Custom patches, upgrade java, new init script
• 7.04– next installer (9.0.505.0)
• 7.32– Patchset
Doing the job
• Zaterdagmorgen– Migration reminders
• Loadbalanced specific aanpassingen• Custom authentication aanpassingen (shib, import/export)• Queued tasks• Permgensize• Wrapper.conf• Nieuwe certificaten• Nieuwe firewall (startup booster)• Nieuw loadbalancer check script• Restore client files• Allowlinking• Sql tuning
• 9.20 => Blackoard release naar toledo team
Nazorg
• Zaterdag– Bijstaan toledo team – performance
• Zondag– 9.00 – 19.30 Verdere afwerking
• Collab server – https• Undotbs, temptbs shrinken• End-user performance tuning (oracle EM)• Deploy on all servers in cluster
– 19.30 – Blackboard available for end users
• Lopend– Upgrade applicatie servers RHEL4 => RHEL5– Upgrade database server 10.2.0.3 => 10.2.04– End user experience (slow)
Performance + bevindingen
• New = cool– +/- 2 weken overload op de servers
• Gemiddeld 300 CPS/ cluster (cluster 4-6 servers)• Na upgrade 250 CPS/ server
Iedereen wil het nieuwe zien
• Garbage collection problemen– Stop the world ...
• Behind.blackboard.com– Informatie, maar niet altijd even duidelijk
• Veel tuning nodig aan de databank
Vragen
• Vragen ?