Development of a Business Intelligence Environment for e-Gov Using Open Source Technologies

T.B. Pedersen, M.K. Mohania, and A M. Tjoa (Eds.): DaWaK 2010, LNCS 6263, pp. 203–214, 2010. © Springer-Verlag Berlin Heidelberg 2010

Development of a Business Intelligence Environment for e-Gov Using Open Source Technologies

Eduardo Zanoni Marques, Rodrigo Sanches Miani, Everton Luiz de Almeida Gago Júnior, and Leonardo de Souza Mendes

Department of Communication, School of Electrical and Computer Engineering, University of Campinas Campinas, SP, Brazil

{emarques,rsmiani,elagj,lmendes}@decom.fee.unicamp.br

Abstract. It has become common for modern organizations to use advanced in-formation systems for helping their daily operational task. However, there is still a large demand for software solutions that enable straightforward data analysis from these systems. Aiming to solve this problem, Business Intelli-gence (BI) environments were created. Electronic Government (e-Gov) sys-tems, which typically work with governmental operational data, can take great benefits from a BI environment. Therefore, in e-Gov systems BI tools can be used, among others, to pursue the following goals: enhance the relationship be-tween city and state government and the citizen; help administrating public re-sources; monitor the impacts of public policies upon the society. This paper presents a proposal for creating a BI environment for Electronic Government systems, using open source technologies with a special application to Social Welfare, developed for the city of Campinas, SP, Brazil.

Keywords: business intelligence, e-government, open source, software architecture.

1 Introduction

With the technological progress of our society, demands for software solutions have been growing in all sorts of areas, both in public and private organizations. Generally speaking, information systems are focused on data storage and processing, which help organizations’ operational sector execute their daily tasks, such as register commer-cial transactions, calculate payroll, register employee and client personal data and so on. However, there is an eminent demand from organizations management sectors for software solutions to help in the processing, analysis and interpretation of their data. This can provide opportunities for better monitoring of their sectors, making projec-tions and finding business deficiencies and/or opportunities. To accomplish this goal, several tools, technologies and solutions were created and aggregated under the con-cept of Business Intelligence (BI) [1].

The creation of a BI environment is a very challenging task, since it requires the im-plementation and concise integration of such technologies as Data Warehouse (DW), Data Mining and Online Analytical Processing (OLAP). Thus, it is fundamental the

204 E.Z. Marques et al.

usage of a methodology and a reference architecture to support the creation of this kind of environment.

Trying to define a generic architecture for this environment, Moss [2] presents a three-layered architecture, shown in Fig. 1. In the bottom of the figure are the data sources from pre-existing systems. These data are transformed by an extraction, transformation and loading (ETL) process, executed by the bottom layer of the archi-tecture, and sent to the middle layer, where the data is stored. Then, data views are exposed to the upper layer, where are the BI applications that allow users to visualize and manipulate data. These applications are provided to users by an interface, like a web site or a web service. The communication between layers is always mediated by a middleware.

Fig. 1. Generic architecture for a Business Intelligence environment [2]

Although the creation of a BI environment is typical of private organizations, e-Gov systems can also benefit from their applications.

This paper presents a proposal for the creation of a BI environment for e-Gov, utilizing open source technologies. As proof of concept, we present the use of this proposal to develop a BI environment to manage Social Welfare for the city of Campinas, SP, Brazil.

The paper is organized as follow: Section 2 presents a discussion about related works, Section 3 brings the technical background of this paper, Section 4 presents the proposal of this paper, Section 5 describes the case study that is being conducted and, closing the paper, Section 6 brings final considerations.

Development of a Business Intelligence Environment 205

2 Related Work

One can find several works describing the construction of a BI environment in various areas.

In [3] a BI environment is created for a supermarket by building a DW using Mi-crosoft Analysis Services. The work applied clustering techniques to define clients profile and discover what types of merchandise most influenced their purchases. In the paper, no information is given on how the BI data will be made available to the users.

In [4] it is presented the elaboration of a BI environment for a life insurance com-pany. This environment has an architecture similar to the one defined in [2], with the adoption of a DW for data storage. The paper does not inform either how the envi-ronment was built or the technologies used.

In [5] it is presented the creation of a BI environment for a Taiwan Internet Service Providers (ISP). The paper defines its own methodology for the BI construction, with an architecture similar to the one defined in [2]. In this environment, an ETL process is applied to the users’ Internet traffic data, with results stored in a DW, using support tools from MS-SQL 2000. Then, clustering techniques are applied to the data to de-fine the Internet users’ usage profile. This data was made available to the users through a Web Site developed for this purpose using Java e Flash technologies.

In Brazil, [6] discusses the creation of a BI environment for the judiciary system. Here, a DW is chosen to store data generated from the ETL process. The DW was built using the methodology proposed in [7]. However, this paper presents neither the environment architecture nor the technologies used.

Regarding DW construction, which is the most common option for data storage in BI environments, there are two works, [8] and [9], treating of health related systems. The first used SAS Data Warehouse Administrator, while the second followed its own methodology using the DBMS MySQL. In [10] the creation of DW for Geographical Information Systems (GIS) and its challenges are discussed. Because all papers fo-cused only in the DW creation, none discuss how data will be provided to the users. In the Brazilian scenario, Mussi [11] presents how the National Agency of Sanitary Surveillance (ANVISA) created its DW by adopting the methodology of [7] and using the DBMS Oracle to store the DW data and the OLAP tool MicroStrategy to make data available to end users.

This paper differs from those mentioned above by presenting: i) all phases for the creation of the BI environment, covering since the planning of the data storage until providing this data to end users; ii) the architecture of the environment; iii) the tech-nologies and tools used in the BI implementation and their integration, highlighting the fact that all technologies and tools used are open source.

3 Technical Background

3.1 Business Intelligence

Created in 1989 by Howard Dresner [12], the Business Intelligence concept defines an architecture and a set of operational, decision support and database systems that, inte-grated, aims at offering to business community easy access to business information [2].

206 E.Z. Marques et al.

As can be seen in Fig. 1, BI can be divided in three main areas: extract, transform and load process (ETL), data storage and tools provided to end users visualize and manipulate data.

The ETL process focus on loading data from the operational data sources to BI data storage. It is developed in three well defined steps [13]:

• extract: in the first step, operational data from the organization, which can be stored in many forms (like in relational, hierarchical or multidimensional DBMS, spreadsheets or emails) is loaded and sent to the next step;

• transform: in this step the data is corrected and suited for the BI through the ap-plication of different operations, like misspelling correction, domain standardi-zation and purging unneeded fields;

• load: final step, where the resulting data from the transform step is stored in the BI data storage.

The most common option for BI data storage is Data Warehouse (DW). The DW was first defined for Inmon [14] as “a subject-oriented, integrated, nonvolatile, and time-variant collection of data in support of management’s decisions”. Although not man-datory, its utilization in a BI is so usual that Almeida [15] and Biere [16] consider BI as the evolution of DW.

Addressing the applications provided to end users, Biere [16] classify them in three groups: traditional query and report, OLAP and Data Mining.

Query and Report tools help common users browse files and DBMS using friendly graphical interfaces, enabling the creation of reports from these data [16].

Defined by Codd [17], OLAP tools offers, through a well defined set of operations, support to visualization and analysis of multidimensional databases. These databases are composed by data cubes, with a set of data cubes being defined as a data mart, that have two basic entities: facts, which are interest items to the organization, like the sales of a product holding related numerical data, like its value and/or discount, and dimensions, where the data that gives context to the fact is stored hierarchically, like date of a sales [13]. The main operations of an OLAP tool are [18]:

• rollup: where the data from the fact is aggregated to an upper level of a defined hierarchy, decreasing data details;

• drill-down: where data from the fact is decomposed to the lower level of a de-fined hierarchy, increasing data details;

• slice and dice: which enables the application of filters to the resulting data; • pivot: that enables re-ordering data.

These two types of tools previously presented focused on simplifying access to BI data. To complement these, Data Mining tools aims at generating intrinsic informa-tion from data stored in the BI. This kind of tools used techniques divided in two groups [19]:

• descriptive: characterize general properties of the data, finding patterns and rela-tions, using techniques like clustering and market basket analysis;

• predictive: composed of techniques like classification, regression and time series analysis, make inferences on the current data to predict future data.

Development of a Business Intelligence Environment 207

3.2 Eletronic Government

An Electronic Government (e-Gov) system is defined by the use of information and communication technologies to develop and offer governmental services and informa-tion through electronic media (usually the Internet) for citizens, enterprises and gov-ernment employees agencies [20].

The construction of e-Gov systems presents particular challenges, depending on the target users of these systems. For this reason, they are divided into five classes [21]:

• Government to Business (G2B): systems developed focusing the improvement of communication between government agencies and enterprises, in such rela-tions as service providing and regulation;

• Government to Citizens (G2C): these systems seek to improve communication between government agencies and the citizens, using initiatives like creating in-formation portals, where citizens can both find governmental information and express their opinions;

• Government to Government (G2G): these systems focus on improving commu-nication between government agencies, enabling them to work together in a simpler and efficient way;

• Government Internal Efficiency and Effectiveness (IEE): these systems seek to improve the internal operation of government agencies;

• Overarching Infrastructure (Cross-Cutting): systems used to facilitate the inte-gration between the systems from all other classes, applying software and hard-ware integration techniques.

There are diverse papers discussing the challenges of creating e-Gov systems, like [22], [23] and [24].

4 Proposal

This paper proposes a technological composition to create a BI environment to e-Gov systems using open source technologies. The architecture used here is based on the one described in [2], being divided in three layers: the ETL layer; the data storage and view providing layer; and the end users applications layer, which are accessed through a web site. A Fig. 2 presents the technological composition proposed.

4.1 ETL Layer

The Talend Open Studio [25], which is a specialized tool for data integration and migration, was adopted to implement the ETL process over the ETL Layer. Through the configuration and composition of its components, it is possible to read data from diverse formats, like Excel, CSV and XML. The tool also enables connecting to rela-tional, hierarchical and multidimensional DBMS. Different types of transformations, like data normalization and denormalization, data joining using key match and string processing can be applied over the data read. After applying the selected transforma-tions, data can be stored in diverse formats, like the ones from the reading step.

208 E.Z. Marques et al.

Fig. 2. Realization of the architecture proposed in [2] using open source technologies

4.2 Data Storage and View Providing Layer

In the Data Storage and View Providing Layer, the data storage is implemented using relational DBMS MySQL [26]. This DBMS was chosen based on its support to vari-ous types of indexes (B-Tree and Hash, for an instance) and fast data load and access, by using MyISAM tables. The communication between Talend and MySQL is done through JDBC middleware, which is a standard to connect Java applications to rela-tional DBMS.

To implement the data view, Mondrian OLAP Server [27] was selected. Mondrian is an OLAP Server written in Java that allows the execution of multidimensional queries, written in Multi Dimensional Expressions (MDX), on relational databases, presenting the results in multidimensional format. It is done by creating a multidimen-sional mapping over the relational data, defining data cubes, where the queries are executed.

The MDX format is a standard defined by Microsoft to execute queries on data cubes, providing an easier and intuitive form to access data from multidimensional databases. In a certain way, MDX is to multidimensional databases what SQL is to relational databases, except that MDX queries return data cubes, while SQL queries return tables. Also, SQL has a Data Definition Language (DDL), which is not in-cluded in current MDX specification. Besides Mondrian, other tools support MDX queries, like MicroStrategy, SAS e Oracle Essbase [28].

Development of a Business Intelligence Environment 209

Using data views brings extra benefits to BI, considering it enables creating differ-ent views to different users and providing a clearer presentation of data to end users.

4.3 End Users Applications Layer

To provide data access to end users, the OpenI [29] application was chosen. OpenI is a web application for executing queries in multidimensional databases. Using OpenI, users can select a data cube and, through a graphical interface, browse data using the main operations of an OLAP client. It is also possible to browse data using MDX to more advanced queries. The resulting data can be displayed in multidimensional ta-bles or charts, like bar and pie. These queries can be saved in the application for fur-ther visualization and/or edition.

Another important OpenI feature is the dashboard creation, where previously saved queries are put together for fast analysis of important data. Thus, it is possible to choose different types of data display for each query result.

The communication between OpenI and Mondrian OLAP Server is done through HTTP, using Simple Object Access Protocol (SOAP) and XML for Analysis (XMLA) protocols.

The SOAP protocol structures information exchange through HTTP. Using XML, this protocol is composed by an envelope which has two tags: header, where infor-mation pertinent to data processing is allocated, and body, that contains the data from the message being transferred [30].

Using SOAP, XMLA protocol defines a couple of XML messages for communica-tion between an OLAP client and server, establishing an interoperable client-server communication channel where diverse tools from different enterprises can work along. The messages defined in this protocol are [28]:

• discover: used to query the OLAP server properties, like a list of available data sources;

• execute: used to execute queries to a specific data source, using MDX language.

5 Case Study

This section describes the application of the proposal to create a BI environment to analyze the operational data from the Social Welfare Department of the city govern-ment of Campinas, SP, Brazil.

The Social Welfare Department has the mission of “rescuing citizenship to people that are in a social vulnerability situation caused by poverty and exclusion”. Thus, diverse social programs have been developed aiming to offer to citizens in vulnerable state new possibilities to enable their financial emancipation. These social programs include differ-ent areas, like offering microcredit, digital inclusion and training courses [31].

To manage data from this area, the Social Welfare module from Integrated System for Municipal e-Gov (SIGM) was adopted. The SIGM is an ERP-like system for managing “all services, citizen records, processes management and relevant data for a city’s administration” [24]. In the Social Welfare module, any data from the social programs developed by the city government can be stored and manipulated as needed.

210 E.Z. Marques et al.

As the first step to construct the BI environment, the utilization of a DW to data storage was chosen. Then, a data-oriented analysis, as proposed in [32], was con-ducted. In this, only operational data sources are considered to define the DW data. This analysis resulted in various candidate data cubes to compose the BI database. From these cubes, the Inclusion in Social Program data cube was selected to be the first offered by the BI.

With the data cube selected, the next step was defining what data would compose this data cube. It was done in two steps. First, a conceptual model was generated (Fig. 3), having the fact, the dimensions, the dimensions hierarchical levels and fields. This model was created in UML using the profile proposed in [33]. Based in this model, the logical and physical modeling was conducted. In this step, the star model pro-posed in [7] was adopted, where the fact is the central element surrounded by its di-mensions. The resulting physical model is illustrated in Fig. 4.

After composing the models, the ETL process was configured using Talend tool. Given the number of treatments needed to be applied in the data before it was inserted on the BI data storage, this step, as predicted in [13], was the one that took the largest development effort in the case study.

Parallel to the ETL step, the DW data view exposed by Mondrian was configured. It was done using the Schema Workbench, an auxiliary tool from Mondrian. In this tool, data cubes are created by defining a fact and the dimensions related to this fact. It is also configured both fact and dimensions fields, which can be from data consoli-dated on the DW or calculated at runtime. The resulting data view is similar to the conceptual model (Fig. 3), which provides a clearer representation of the BI data to its users.

Fig. 3. The Inclusion in Social Program data cube conceptual model

Development of a Business Intelligence Environment 211

Fig. 4. The Inclusion in Social Program data cube physical model

Fig. 6. The OpenI OLAP Client interface, with a query resulting data displayed in a dimen-sional table.

Once both ETL and data view steps were concluded, the next step was to configure

OpenI to access the data through the data view exposed in Mondrian. This configura-tion is done using its administrator interface, where the data sources are registered. After registering the data source, it was already possible to access the BI data using OLAP queries, which were created using OpenI OLAP Client interface, as can be seen in Fig. 6. It is important to say that the data exhibited in the figure does not cor-respond to the real data from the BI constructed.

212 E.Z. Marques et al.

The creation of this BI environment, although having only one data cube, enabled the discovery of diverse information about inclusions in social programs, like periods when the highest inclusion rate happens for a specific social program and major char-acteristics of citizens included in a specific social program. This information is very useful to analyze the execution of social programs, considering factors like its effec-tiveness.

6 Conclusion

This paper presented a proposed technological composition to create a BI environ-ment to e-Gov systems using open source technologies. It also presents the applica-tion of this proposal to develop a BI environment to manage Social Welfare for the city of Campinas, SP, Brazil.

The creation of the BI environment in the case study was done through the follow-ing steps:

i. analysis of candidate data cubes to compose the DW and selection of one to be implemented;

ii. conceptual, logical and physical modeling of the chosen data cube; iii. configuration of the ETL process in Talend tool; iv. creation of the data view in Mondrian OLAP Server; v. configuration of OpenI to access the data view exposed in Mondrian, provid-

ing users access to data.

The case study is in its beginning, having still a few requirements not being consid-ered, like data access security and queries performance. Nevertheless, analyzing the data already provided to end users, we could conclude that the case study validates the proposal.

As future work, besides the mitigation about the requirements not considered and the inclusion of new data cubes in the BI, will be conducted a study analyzing data mining techniques that can be applied in this scenario. It is also planned a deeper analysis of the impacts of applying BI environments to e-Gov.

Acknowledgments. Rodrigo Sanches Miani's work is supported by the State of São Paulo Research Foundation (FAPESP).

References

1. Xu, L., Zeng, L., Shi, Z., He, Q., Wang, M.: Research on Business Intelligence in enter-prise computing environment. In: ISIC. IEEE International Conference on Systems, Man and Cybernetics, pp. 3270–3275 (2007)

2. Moss, L.T., Atre, S.: Business Intelligence Roadmap: The Complete Project Lifecycle for Decision-Support Applications. Addison-Wesley Longman Publishing Co., Inc., Amster-dam (2003)

3. Hong, X., Zai-wen, L., Hai-yang, M.: Study and Realization of Supermarket BI System Based on Data Warehouse and Web Technique. In: CSSE 2008: Proceedings of the 2008 International Conference on Computer Science and Software Engineering, pp. 482–485. IEEE Computer Society, Washington (2008)

Development of a Business Intelligence Environment 213

4. Xu, Z., Zhang, M., Jiang, X.: Business Intelligence - A Case Study in Life Insurance In-dustry. In: ICEBE 2005: Proceedings of the IEEE International Conference on e-Business Engineering (2005)

5. Li, S., Shue, L., Lee, S.: Business intelligence approach to supporting strategy-making of ISP service management. J. Expert Syst. Appl. 35, 739–754 (2008)

6. Ruschel, A.J.: Governo eletrônico: Business Intelligence para a modernização do Ju-diciário (2008)

7. Kimball, R.: The data warehouse toolkit: practical techniques for building dimensional data warehouses. John Wiley & Sons, Inc., Chichester (1996)

8. Sahama, T.R., Croll, P.R.: A data warehouse architecture for clinical data warehousing. In: ACSW 2007: Proceedings of the Fifth Australasian Symposium on ACSW Frontiers, pp. 227–232. Australian Computer Society, Inc., Australia (2007)

9. Wah, T.Y., Sim, O.S.: Development of a data warehouse for Lymphoma cancer diagnosis and treatment decision support. J. WSEAS Trans. Info. Sci. and App. 6, 530–543 (2009)

10. Rifaie, M., Blas, E.J., Muhsen, A.R.M., Mok, T., Kianmehr, K., Alhajj, R., Ridley, M.J.: Data warehouse architecture for GIS applications. In: iiWAS 2008: Proceedings of the 10th International Conference on Information Integration and Web-based Applications & Services, pp. 178–185. ACM, New York (2008)

11. Mussi, C., Murahovschi, D., Bettni, G., Kratz, L.G.: Data Warehouse - A Experiência da ANVISA. In: IX CBIS - Congresso Brasileiro de Informática em Saúde (2004)

12. Business intelligence at age 17, http://www.computerworld.com/s/article/266298/BI_at_age_17

13. Kimball, R., Reeves, L., Ross, M., Thornwaite, W.: The Data Warehouse Lifecycle Tool-kit: Expert Methods for Designing, Developing and Deploying Data Warehouses. John Wiley & Sons, Inc., Chichester (1998)

14. Inmon, W.H.: Building the Data Warehouse. John Wiley & Sons, Inc., Chichester (1992) 15. Almeida, M.S., Ishikawa, M., Reinschmidt, J., Roeber, T.: Getting Started with Data

Warehouse and Business Intelligence. IBM Redbooks (1999) 16. Biere, M.: Business intelligence for the enterprise. IBM Press (2003) 17. Codd, E.F., Codd, S.B., Salley, C.T.: Providing OLAP to User-Analysts: An IT Mandate

(1993) 18. Chaudhuri, S., Dayal, U.: An overview of data warehousing and OLAP technology. J.

SIGMOD Rec. 26, 65–74 (1997) 19. Dunham, M.H.: Data Mining: Introductory and Advanced Topics. Prentice Hall PTR,

Englewood Cliffs (2002) 20. Yu, C.: Role-Based and Service-Oriented Security Management in the E-Government En-

vironment. In: EGOV 2009: Proceedings of the 8th International Conference on Electronic Government, pp. 364–375. Springer, Berlin (2009)

21. Lee, S.M., Tan, X., Trimi, S.: Current practices of leading e-government countries. J. Commun. ACM 48, 99–104 (2005)

22. Velsen, L., Geest, T., Hedde, M., Derks, W.: Engineering User Requirements for e-Government Services: A Dutch Case Study. In: Wimmer, M.A., Scholl, H.J., Ferro, E. (eds.) EGOV 2008. LNCS, vol. 5184, pp. 243–254. Springer, Heidelberg (2008)

23. Al-Omari, H.: E-Government Architecture in Jordan: A Comparative Analysis. J. of Com-puter Science 2(11), 846–852 (2006)

24. Tilli, M., Panhan, A.M., Lima, O., Mendes, L.S.: A Web-based Architecture for e-Gov. Application Development. In: ICE-B: Proceedings of the International Conference on e-Business (2008)

25. Talend: Open Source ETL and Data Integration Software, http://www.talend.com

214 E.Z. Marques et al.

26. MySQL, http://www.mysql.com 27. Mondrian, http://mondrian.pentaho.org 28. XML for Analysis, http://www.xmla.org 29. OpenI: Open Source Business Intelligence for On-Demand Deployments,

http://openi.org 30. Snell, J., Tidwell, D., Kulchenko, P.: Programming Web services with SOAP. O’Reilly &

Associates, Inc., Sebastopol (2002) 31. Prefeitura Municipal de Campinas,

http://2009.campinas.sp.gov.br/trabalho/ 32. Inmon, W.H.: Building the Data Warehouse, 4th edn. John Wiley & Sons, Inc., Chichester

(2005) 33. Luján-Mora, S., Trujillo, J., Song, I.: A UML profile for multidimensional modeling in

data warehouses. J. Data Knowl. Eng. 59, 725–769 (2006)