Hosting Millennium Project AIUG Lesley Sharp Manager, Online Strategy 16 November 2012.

Hosting Millennium ProjectAIUG

Lesley SharpManager, Online Strategy

16 November 2012

Background

SA Government StateNet

SLSA

Issues - 2008-2009

• Millennium server• 7 years old and needed to be replaced• Not compliant with SA Government Security

requirements (database and web server on same server)

• DMZ environment• Even if the web serving module was deployed and

application and database servers separated, approval for SSH access would be refused

• III Support Requirements vs StateNet Support Requirements• III preferred support method = SSH (Port 22)• StateNet = VPN, two-factor authentication and no

split tunnelling• Testing showed that two-factor authentication process

raised other issues for III, and couldn’t realistically be achieved (clash of IP numbers)

INTERNETINTERNET

PATRON

SLSASLSA DMZA

DMZAIII

IIIDMZA

DMZA

SLSASLSA

HSP Co-located

Hosting

HSP Co-located

Hosting HSP Dedicated

Hosting (VS)

HSP Dedicated

Hosting (VS)

HSP Dedicated Hosting (VS)

HSP Dedicated Hosting (VS)

INTERNETINTERNET

STAFF

1. 2. 4.3. 5. 6. 7.Role

Based DMZ

Role Based DMZ

8.

Option 4

• October 2009 – commenced discussions with III about their willingness to host our Millennium system in Emeryville

• Commenced a one month trial from early December 2009 of hosting Millennium with III

Show-stopper

• Serial check-in card!

• Drawing the 84-box check-in card on the screen consistently took 20 seconds

• III staff testing in Emeryville – instantaneous

• III staff tested from Ohio and the response time reduced to 9-14 seconds

• Investigations with Dimension Data to check if packet-shaping of data through StateNet was causing this problem

Show-stopper cont’d

• Tested from different parts of the network with no change in the delay time

• Dimension Data traced the data and advised there was no packet loss and the delay was caused by latency while traversing networks and/or the application itself may be the cause

• End result that Hosting Millennium with III in the USA was not a viable option

• Project closed 23 March 2010

Option 5 – Co-Located Hosting

• SA Government Hosting Panel established 2008

• SLSA issued RFQ in June 2010 for “Un-managed Co-Location” services i.e. Place our new server in rented rack space outside of StateNet and have SA Government DCSS vendor manage the server for us

• No formal response to this RFQ, general lack of rack space in Adelaide at that time

Option 6 – Virtual Server - Hosted

• 9 July 2010 – Issued RFQ for a Virtual Private Server in a dedicated hosting environment

• One response – from Internode

• 19 August 2010 – Customer Agreement established

• 16 September 2010 – Virtual Private Server commissioned

• Hooray – all our problems are over!

Migration

• Opened a new project with III, engaged a project manager

• Held a series of planning teleconferences

• Migration date of 26 October 2010 agreed

• Outage notifications were issued a week prior to staff and customers – advising up to 72 hours outage, with possibility of early release

Issues

• Required to convert from Method I indexing to Method II indexing for Advanced Word Searching before the migration could occur

• III Project Manager unavailable

• III had forgotten that the data would need to be converted from Sun to Linux

• Release 2009A and Linux 64 bit create performance problems

• Unplanned immediate upgrade to Release 2009B• Caused issues with acquisitions data

Issues cont’d

• Firewall not set up correctly, needed changes

• Host name not set up correctly, not fully qualified

• Issues with testing the Enterprise Backup API

• Communications time lag between USA and SA

• SLSA Information Desk staff provided access to the catalogue despite an official “outage” still being in place

Issues cont’d

• When testing commenced, not all staff had the most recent Millennium client

• Staff did not follow the test plan• Other than seeing how quickly the serials check-in

card rendered

• Found it was impossible to get the required number of people testing at the same time

• We were too far down the track to consider rolling back

Issues cont’d

• And as we segued from testing to normal operations we started to encounter

Performance Issues

Disconnections

Issues cont’d

• PC power settings adjusted

• Higher than acceptable network collision rate

• Commumications cupboard re-cabled

• Millennium clients freezing; response times slow

• Upgraded server with memory and CPU

• Ongoing performance issues

• III check VMWare settings

Issues cont’d

• Internode add VMWare Tools

• Internode apply Resource Groups

18 February 2011

Performance issues resolved

Issues cont’d

• Disconnections• Dimension Data - Packet tracing

• StateNet Intrusion Prevention System – change required to bypass rules

10 March 2011

Disconnect problem declared ‘solved’

Other issues

• 16 March 2011• Kernel problems commence• Related to install of Encore Reporter?• Resolved by booting into “old” kernel

• 25 March 2011 (a Friday of course)• Performance issues – slowness and Server Not

Responding messages• This turns out to be caused by an incident at Internode,

unrelated to anything else (phew!)

• No other outages caused by hosting environment

Benefits of hosting

No more StateNet!

Better backup regime

Ability to increase server resources quickly and simply

III can provide support using their preferred method

Have been able to deploy our backlog of modules (RSS, WAM, Encore, Encore Reporter, ERM, Content Cafe)

Can now plan upgrades such as to Sierra with more confidence

Thank you

Lesley SharpManager, Online StrategySharp.lesley@slsa.sa.gov.au

State Library of South Australia, North Terrace, Adelaide SA 5000Phone: (08) 8207 7250 - www.slsa.sa.gov.au