Implementing a Outsourced Technology Infrastructure

- 1 -

Implementing a Outsourced Technology Infrastructure

SoutheastCon 2007 22-25 March, 2007

AJ Burke Chief Technology Officer

(2001-2005) HASCO America [email protected]

§ Copyright 2007 by A.J. Burke. Released for use by the IEEE SoutheastCon 2007, for publication in all forms, with permission.

- 2 -

1.0 Introduction

1.1 Thesis 1.2 Business Example

2.0 Strategic Planning

2.1 Plan Updates 2.2 Strategic Goals

2.3 New Operational Concepts

3.0 Detailed Design

3.1 IPSEC Virtual Private Network 3.2 Network partners

4.0 Implementation

4.1 Systems Architecture 4.2 Hosting Providers

4.3 Open Source Solutions

5.0 Staff Development

5.1 Help Desk Training

5.2 Network Operating Systems

5.3 Applications Training

6.0 Conclusion 7.0 References

- 3 -

1.0 Introduction 1.1 Thesis Modern businesses need significant telecommunications and computing capability to succeed in these days of extreme cost pressure due to globalization. To be successful, it is often necessary to use other business trends like outsourcing of non-critical and, sometimes, even mission-critical functions to meet cost requirements to be competitive in the global market. This paper will illustrate some of the planning, design, implementation and staff development issues related to one such a strategy in a multi-year project designed to implement a new enterprise architecture in the North and South America region as part of a larger global effort (Europe, America, Asia). This effort involved realigning staff and external contract resources to match a new business model based upon a strategy to use the services now provided by the public Internet. 1.2 Business Example In the late 1990’s, a European company involved in the Plastics business moved to consolidate its operations in the Americas region from a distributor-based model to a single company business model by acquiring its distributors through as series of buyouts. These distributors operated on a variety of computer and telecommunications platforms and had various levels of Internet access from dial-up to lower bandwidth ‘high-speed’ solutions like ISDN. In addition, as this business effort continued the opportunity to outsource many technology functions and requirements bean to be realized. This strategy required significant training of engineering, finance, materials and other corporate staff on Internet threats and security procedures to be able to conduct routine engineering and business operations over the public Internet safely.

Figure 1. HASCO1 global presence in IEEE regions

- 4 -

Of course, the Internet and its associated technologies are always the real medium of interest here in the beginning of the 21st century. But being able to move critical operations like order management, inventory management, accounting and engineering design to rely solely on Internet protocols and services is still a somewhat elusive goal in some corporate environments. One specific technology used to implement this architecture on the public Internet, after some initial implementation steps using the traditional dedicated T1/E1 private networks in the Americas region that proved too expensive, was Internet Protocol Security (IPSEC) Virtual Private Network’s (VPN’s) over an expansive America-wide wide network. This architecture also included the use of Open Source (Linux) solutions for distributed file system management, network monitoring and intrusion detection. Some of the Service Oriented Architecture (SOA) enterprise services used to delivered the elements of the enterprise architecture discussed include the movement from 2-tier to N-tier applications deployment models and elements of core of the original Internet protocols that still find extensive use in engineering applications like RPC, FTP and NFS. Finally, providing the Internet firewall and VPN secure technologies mentioned above using first tier Internet providers and security enhancement with Open Source network Intrusion Detection and Protection Systems (IDS/IPS) is discussed.

2.0 Strategic Planning This section illustrates the elements of strategic planning that where conducted to maintain the linkage of the business plan with the technology infrastructure and support requirements to accomplish it. This was effectively a series of management and staff meetings that set up the high-level goals and objectives of the effort. 2.1 Plan Updates

Figure 2. Standard Technology Steering Committee opening slide

- 5 -

Routine annual, and sometimes semi-annual, strategic planning updates provided to the HASCO America Board of Directors or senior staff where used as an opportunity to acquire feedback on the goals and direction for technology infrastructure efforts. These always included stating, and sometimes re-stating, assumptions made and current goals and objectives. Often new technical and operational procedures and techniques where presented during these sessions also. Finally, information updates based upon statistical analysis of operational status (Help Desk trend data) was provided to show the on-going operational workload and any impact, if any, created by infrastructure configuration changes. 2.2 Strategic goals An example slide from one of these presentations is given here. The overarching goal to consolidate and simplify all Internet access was seen as a way to get the necessary focus on the network management, help desk, operating procedures, software standardization and enterprise planning and execution systems. This was predicated on assumptions that any system design would comply with existing company standards for computer and telecommunications equipment and software, that existing systems would be supported until the alternatives where implemented and that there was overall management support for the standardization of systems. Eventually cost reduction pressures forced the reevaluation of the idea that existing systems could continue to be supported at the Help Desk and helped in justifying the retirement of some of these obsolete environments.

Figure 3. Another standard slide stated that stated goals and objectives

- 6 -

2.3 New operational concepts As new operational modes where investigated, such as VPN versus the traditional phone system T1/E1’s and outsourcing of local network operations support to qualified suppliers, the management team was introduced to these ideas during strategic Technology Steering Committee sessions. This allowed for better communications with the management and staff as the eventual changes where made in the infrastructure supporting their business needs and reduced the amount of confusion that these changes caused in ongoing business operations.

Figure 4. New technology infrastructure concept proposal

3.0 Detailed Design This section will show some elements of the more technically and operationally interesting aspects of the work that was accomplished in moving this infrastructure along through its various levels of development over a 5 year period. First the conceptual view of the new VPN network is expanded with some detail (note that IP information has been sanitized in the graphic) procedures and the network structure as actually used in the design.

- 7 -

3.1 IPSEC Virtual Private Network (VPN)

Figure 5. More detailed information on possible VPN tunnels

When this infrastructure was first started some of the locations had dial-up and ISDN access to the Internet. This was replaced by standard business-class T1/E1 (1.544 MBPS/2.048 MBPS) access including international long-distance links that gave the developers and administrators more robust capabilities. When the expense of these dedicated international T1/E1’s was later found to be limiting, they where replaced by the Internet Protocol Security (IPSEC) Virtual Private Network (VPN) still in use today in most locations. This allowed for the local T1/E1 business-class connections that are much less expensive. Instead of the proprietary firewall/VPN hardware selected by corporate, enough latitude was found within the corporate standard to allow for hosting on common off the shelf server platforms that greatly reduced the cost of the deployment.

3.2 Network providers It is important to have good first-tier wide area network providers in all geographical areas when outsourcing non-critical and critical business processes. This when sometimes even picking a network service provider for residential service when moving to a new area is no small task! In this implementation corporate standards required the vendor, but key and critical to the success of the effort in the America region was early liaison with the supplier (MCI at that time) Network Operations Center (NOC) to insure that the needs within and eventually between the geographic regions where met.

- 8 -

Figure 6. Primary network provider2.

4.0 Implementation Elements of the implementation required internal architecture details at a bit more of level of detail to be defined. We will look at the logical block diagram that defines the internal system architecture, define the hosting environment and discuss some interesting open source solutions to apply to the infrastructure. This section will provide some details of this activity. 4.1 Systems Architecture Aging IBM minicomputers in international locations where replaced with commodity server platforms in outsourced central data centers with whatever standard hardware the hosting provider supplied. Since some customization of the interface architecture using commercial firewall software was specified by company global IT standards, this capability was a requirement in picking these suppliers. A common distributed file system based on Linux NFSv4 (Network File System version 4) was implemented. NFSv4 is a distributed file system protocol based on NFS protocol versions 2 [RFC1094] and 3 [RFC1813]3&4. Unlike earlier versions, the NFS version 4 protocol supports traditional file access while integrating support for file locking and the mount protocol. In addition, support for strong security (and its negotiation using strong authentication), compound operations, client caching, and internationalization have been added. Attention has also been applied to making NFSv4 operate well in an Internet distributed environments. HASCO_COMMON was built with NFSv4 and made globally available for use within the hemisphere region internally and included external access for the customer at network address of ftp://ftp.us.hasco.com.

- 9 -

Figure 7. Logical diagram of internal structures and functions

4.2 Hosting Providers Hosting providers where selected from the top tier of web service providers that also allowed for some internal hardware and software customization of their servers for specific backend, mission-critical order processing. Eventually, all inventory management, accounting and many elements of engineering design where moved into the hosted environment as well. Not since the early days of the time-shared mainframe has the concept of total reliance on such outsourced arrangements been conceivable by many business organizations. But with the critical nature of website uptime and availability being what it is for many on-line companies these days, this is now a much more achievable goal. Of course, sufficient backups and other disaster recovery and business continuity needs must be met in some fashion. They where in this architecture as well, but those issues involving the data warehouse implementation that came into being along with other system elements (specifically the ERP/CRM components), but that is a bit beyond the current scope. 4.3 Open Source Solutions

Figure 8. Open source has its own market presence also!

- 10 -

Enhanced distributed file systems, network operations and security using Open Source (Linux) solutions was added after the expansion of the engineering systems. These included the Linux operating system with its built-in Network File System (NFS) distributed file system, the Multi Router Traffic Grapher (MRTG) and the Snort Intrusion Detection/Prevention System (IDS/IPS).

Figure 9. Internet uptime and traffic was monitored using Open Source

The Multi Router Traffic Grapher (MRTG) is a tool to monitor traffic load on network links. MRTG generates HTML pages containing images which provide a visual representation of this traffic. MRTG works on most UNIX/Linux platforms and Windows. MRTG is written in Perl and comes with full source5.

Figure 10. Intrusion Detection and Prevention was implemented with Open Source too

Snort is considered by some to be a ‘lightweight’ intrusion detection system that was developed around 1998 by Martin Roesch and others. But that has changed greatly as Snort has become perhaps the most successful of the IDS/IPS environments due to its open source nature and continued development and support by the open source community6.

- 11 -

5.0 Staff Development No predetermined level of computer system knowledge was assumed for anyone during this implementation and everyone from the CxO level on down had to sit for or test out of required training on routine desktop maintenance and operations, applications and architectural elements of the environment. This extensive focus on training paid off in assuring that Help Desk personnel had and could use those training records. And this was the case on many occasions with Help Desk calls that came in. The Help Desk also allowed for escalation procedures to external contract resources and access by those personnel to assist the first responders. 5.1 Help Desk Training Early in the effort a consolidated regional Help Desk was established with web capabilities. This was used for tracking all operational and deployment issues as well as providing an underlying database for statistical analysis of trends in the technology operations data. This was a critical feedback mechanism into the planning mentioned above as well.

Figure 11. Critical issues where captured in the America regional Help Desk

- 12 -

5.2 Network Operating Systems

The primary operating environment for desktops during the period of the infrastructure development was the Windows 2000 Professional system. All the training conducted assumed that this was the network operating environment, although early on there where still Windows NT machines and later on Windows XP appeared on some desktops. A typical introductory training session included security measures like network logon/logoff procedures, simple operations like adding shortcuts to manipulate the desktop, majors components of the environment, new capabilities, use of help and Internet based software update procedures for the operating systems and desktop virus, firewall and intrusion prevention systems.

Figure 12. Boot Camp for executives, sales, finance, distribution and engineering

Individuals where assumed to be capable to perform routine maintenance of their desktops after the training. Help Desk personnel would often direct them to certain parts of those training materials to refresh them on desktop procedures. This was typical when Help Desk work orders related to slow performance or other similar issues when reported. If these routine procedures did not improve the situation then other measures such as dispatch of a local country (Canada, US, Mexico) technician could be accomplished with the Help Desk systems also. Plus second tier systems engineering personnel where available on a outsource basis and could be brought into a situation via external Internet access to the Help Desk system.

- 13 -

Figure 13. Many routine maintenances where accomplished by the user.

5.3 Applications Training

Figure 14. Some externally supported applications

- 14 -

Finally a few of the Internet portal based environments for customers and internal engineering applications are provided here for completeness in showing the real underlying reasons for the extensive effort expended on this infrastructure implementation. Computer Aided Design/Manufacturing and Engineering services along with customer service and internal operations management where enhanced tremendously by the architecture.

Figure 15. Some internally supported engineering applications

- 15 -

6.0 Conclusion This paper has shown how significant telecommunications and computing capability can be accomplished, even in industry situations where extreme cost pressures due to globalization are present, using outsourced providers and open source to provide effective solutions. It can also be seen how initially outsourcing non-critical functions can eventually be used to do the same with mission-critical functions. The example here is some of the planning, design, implementation and staff development issues related one such strategy in a multi-year project designed to implement a new enterprise architecture in the North and South America region as part of a larger global effort (Europe, America, Asia). To accomplish this it was necessary to realigning technology staff internally to focus on being effective first responders on a hemisphere regional Help Desk and how to leverage external contract resources deal with other elements of support. This might be just to dispatch a technician that is physically located near a supported facility or involve second tier of support that is more qualified to handle specific issues such as database or network administration. In either case access was accomplished with by using the system itself for access via Internet. This strategy also requires significant training of non-technology staff in engineering, finance, materials and other corporate staff on Internet threats and security procedures to be able to conduct routine engineering and business operations over the public Internet safely. The ability to move critical operations like order management, inventory management, accounting and engineering design to rely solely on Internet protocols and services can be achieved. Specific technologies that can be used to implement this type of architecture may vary, but most will depend on at least some of the public Internet technologies such as IPSEC VPN’s and other Service Oriented Architecture (SOA) enterprise services. Open Source (Linux) solutions for distributed file system management, network monitoring and intrusion detection as well as core of the original Internet protocols that still find extensive use in engineering applications like RPC, FTP and NFS may also be effective in implementing this type of infrastructure and saving on software licensing costs.

- 16 -

7.0 References 1. HASCO Hasenclever GmbH. http://www.hasco.com 2. MCI. http://www.mci.com 3. Linux Online. http://www.linux.org 4. Linux Documentation Project. http://www.tldp.org 5. Multi Router Traffic Grapher. http://www.mrtg.com 6. Snort. http://www.snort.org