-
. . - .
INFORMATION AND TECHNOLOGY TO ENABLE YOUR BUSINESS
I T @ W B S T A F F W O R K I N G P A P E R S
Impact Assessment Study of Computerized Services Delivery
Projects from India and Chile
Primary author:
Subhash Bhatnagar Professor, Indian Institute of Management,
Ahmedabad
Contributors to report:
Jiro Tominaga Senior lnformation Officer, ISG E-Government
Practice
Shirin Madon lnformation Systems & Innovations Group, Dept.
of Management, London School of Economics
Deepak Bhatia Manager, ISG E-Government Practice
Pub
lic D
iscl
osur
e A
utho
rized
Pub
lic D
iscl
osur
e A
utho
rized
Pub
lic D
iscl
osur
e A
utho
rized
Pub
lic D
iscl
osur
e A
utho
rized
Pub
lic D
iscl
osur
e A
utho
rized
Pub
lic D
iscl
osur
e A
utho
rized
Pub
lic D
iscl
osur
e A
utho
rized
Pub
lic D
iscl
osur
e A
utho
rized
-
The findings, interpretations, and conclusions expressed in this
book are entirely those of the authors and should not be attributed
in any manner to the World Bank, to its affiliated organizations,
or to members of its Board of Executive Directors or the countries
they represent. The World Bank does not guarantee the accuracy of
the data included in this publication and accepts no responsibility
for any consequence of their use. The boundaries, colors,
denominations, and other information shown on any map in this
volume do not imply on the part of the World Bank Group any
judgment on the legal status of any territory or tho nnrlnrrnmnnt n
r =rrnnt=nrn nf cl orh hnnonrl=r;nr
-
IMPACT ASSESSMENT STUDY
OF
COMPUTERIZED SERVICE DELIVERY PRO JEClTS
FROM INDIA AND CHILE
June 2007
-
Study Coordinator:
Task Managers:
Study Team:
Indian Institute of Management, Ahmedabad (LIMA)
London School of Economics
Partners:
Subhash Bhatnagar, Adjunct Professor, Indian Institute of
Management, Ahmedabad (IIMA)
Deepak Bhatia, Manager, e-Government Practice, Information
Solutions Group (ISGEG) Jiro Tominaga, Senior Information Officer,
ISGEG
T. P. Rama Rao, Professor Nupur Singh, In-charge, Center for
Electronic Governance Ranjan Vaidya, Research Associate, Center for
Electronic Governance Mousumi Mandal, Research Assistant, Center
for Electronic Governance
Shirrn Madon, Senior Lecturer, Information Systems and
Innovation Group, Department of Management Matthew Smith, Doctoral
Student, Information Systems and Innovation Group, Department of
Management
The World Bank: ISGEG IIM A Ministry of Communications and
Information Technology, Government of India
-
ABBREVIATIONS AND ACRONYMS
AMC AP
B2C CARD
C-DAC
DFID eGEP EU
G2C ICT
r n A IT KAVERI
LSE
MIS OECD PRM
RTC SII
UN UNCTAD
LTNDP UNPAN
Ahmedabad Municipal Corporation Andhra Pradesh
Business to citizen Computer Aided Administration of
Registration Department
Centre for Development of Advanced Computing
Department for International Development
eGovernment Economics Project European Union Government to
citizen Information and Communications Technologies
Indian Institute of Management, Ahmedabad
Information Technology Karnataka Valuation and
e-Registration
London School of Economics and Political Science Management
Information Systems
Organization for Economic Co-operation and Development
Performance Reference Model Records of Rights, Tenancy and Crop
Inspection Register
Servicio de Impuestos Internos
United Nations United Nations Conference on Trade and
Development
United Nations Development Programme United Nations Online
Network in Public Administration and Finance
-
This research is the product of a collaborative effort on the
part of the faculty of the Indian Institute of Management,
Ahmedabad (IIMA) and the London School of Economics, as well as the
staff of the Information Solutions Group of the World Bank. It
benefited greatly from a series of consultations with internal and
external experts. A number of versions of the concept note for the
study were circulated to solicit input and comments. The authors
would like to extend special thanks to: Bruno Laporte; David
Wheeler; Pierre Guislain; Philippe Dongier, Christine Qiang,
Randeep Sudan, and Isabel Neto; Mark Dutz; Markus Goldstein and
Sumeet Bhatiti; Soren Gigler; Gregory Kisunko; Michael Kubzansky;
Jens Kristensen; and Edwin Lau (OECD e-Government Project) for
their valuable suggestions.
This study would not have been possible without the support of
the management of Information Solutions Group, in particular,
Rakesh Asthma, senior manager with ISGEC. Deepak Bhatia, in
addition to task management, played a key role in harnessing
support for the study from different units in the Bank.
The entire project team from IIMA, London School of Economics,
and World Bank contributed to the conceptualization of the study
and development of the assessment framework that has been proposed
in the report. The team based at IIMA designed the survey
instruments and sampling methodology, and carried out the impact
assessment for five projects in India. The survey of clients in
India was done by TNS India, one of the large market research
agencies in the country. The survey in Chile was conducted by
Matthew Smith under the guidance of Shirin Madon.
This report is primarily authored by Subhash Bhatnagar. Nupur
Singh and Ranjan Vaidya provided the data analysis support and
generated all the tables presented in the report. Shirin Madon did
a separate paper on literature search, parts of which have been
used in the section that discusses efforts to assess public sector
projects. Jiro Tominaga and Deepak Bhatia worked closely with
Subhash Bhatnagar from the initial concept stage and made
significant contributions in developing the structure and content
of the report.
The research team would like to acknowledge the help provided in
data collection from the agencies and users by all the staff of the
implementing agencies of the projects that were assessed. In
particular, the team is grateful to the following persons: Mr.
Rajeev Chawla, Secretary (e-Governance), Department of Personnel
and Administrative Reforms, Government of Karnataka (GOK); Mr.
Gurumurthy, Deputy Tahsildar, Bhoomi Monitoring Cell, GOK; Mr. S.
M. Bhavikatti, Assistant Inspector General of Registration
(Computer), GOK; Mr. V. N. Thippanagouda, District Registrar and
Deputy Commissioner of Stamps, Bangalore (Urban); Mr. P. Raghuveer,
Additional Secretary, Government of Andhra Pradesh (GOAP); Mr. A.
Raveendranath, Assistant Inspector General (CARD), Department of
Registration and Stamps, GOAP; Dr. N. Sailaja, Special Officer,
Department of IT and Communications, GOAP; Mr. Bireshwar Das,
Deputy Director (Technical), eSeva; Mr. Anil Mukim, Commissioner,
Ahmedabad Municipal
-
Corporation (AMC); Capt. Dilip Kumar Mahajan, Deputy Municipal
Commissioner (Finance and e-Governance), AMC; and Mr. A. P. Patel,
EDP Officer, Computer Department, AMC.
The team would also like to thank: Alejandra Barraza and Mathias
Klingenberg, both working with the Directorate of Public
Procurement Contracting; Fernando Barraza and Victor Hugo Lopez at
Servicio de Impuestos Internos (SII); Viktor Aguila, Mabel Verdi,
and Juan Enrique Hinostroza from the Universidad de la Frontera in
Temuco; and Carolina Seward.
-
In extended consultations, a number of versions of the concept
note for the study were shared with relevant World Bank staff
members since November 2005. The concept note was also reviewed at
the Concept Review Meeting held on January 23,2006.
Originally, the study was to cover five projects in India, two
in Chile and five projects from other countries where assistance
from the World Bank field staff might be sought. Potential projects
were identified on the basis of publicly available evaluation data
and documents. A number of World Bank staffers were contacted to
explore if additional data (client surveys) could be collected.
However, the effort required appeared at that point to be more than
what could be handled. The study team, therefore, decided to limit
the scope of this study to seven projects in India and Chile,
leaving more comprehensive scope for future studies.
The field work was conducted from April 2006 to June 2006. In
May 2006, interim results from the study were discussed with Bank
staff that had provided input on the concept note in meetings held
in Washington, D.C. On May 14, 2006, the interim findings were
presented in a public forum organized by Information Solutions
Group.
Although with limited scope, this study contains analyses of
diverse computerization projects implemented by public sector
organizations to improve service delivery in two developing
countries. The study demonstrates that it is possible to estimate
direct and indirect benefits to clients with reasonable accuracy
using the proposed framework. Our analyses are based on in-depth
surveys of clients (citizens and businesses that use the systems)
and other stakeholders involved in these projects. While there is
room for further refinement, the study makes a modest contribution
in understanding the potential and limitation of the power of
information and communications technologies (ICT) in improving
delivery of services by public sector organizations. The results
were both encouraging and sobering. The potential of ICT to help
reduce cost of accessing services by clients, improve convenience,
further accountability, and promote good governance in a broad
sense is difficult to ignore. However, for those of us in the field
of development assistance, the study also guides us to look beyond
the hype of "e-government." Lack of effective project design,
inadequate infrastructure, and a lack of willingness to reengineer
processes can result in wasted opportunities, with the computerized
systems showing hardly any improvements over the manual systems
that are replaced. Familiar principles of project design and
project management continue to be important for generating
results.
The authors would like to extend sincere gratitude to all those
who supported the completion of this study. It is our hope that
this study becomes a starting point for many new studies to
come.
-
STUDY TEAM
...............................................................................................................................................
I1
...................................................................................................
ABBREVIATIONS AND ACRONYMS I11
ACKNOWLEDGMENTS
...........................................................................................................................
IV PREFACE
...............................................................................................................................................
VI EXECUTIVE SUMMARY
...........................................................................................................................
1 INTRODUCTION AND CONTEXT
............................................................................................................
5
................................................ CHAPTER 1:
ASSESSMENT OF PUBLIC SECTOR ICT PROJECTS 8 CHAPTER 2: MEASUREMENT
FRAMEWORK AND DATA ANALYSIS .......................................
12
..................................................................................
MEASUREME~T F'RAMEWORK AND IV~ETHODOLOGY 12 ANALYSIS OF SURVEY DATA
..................................................................................................................
16
.................................................................................................................
1 . Pro$le of respondents 16
................................................................................................
2 . Dimensions of impacts on clients 16
....................................................................................
3 . Analysis of data collected from agencies 17 CHAPTER 3:
ASSESSMENT RESULTS FOR INDIAN AND CHILEAN PROJECTS
..................... 18
..................................................... A
COMPARATIVE ANALYSIS OF PROJECTS FROM CLIENT PERSPECTIVE 18 1 .
Cost to users
..............................................................................................................................
18
..............................................................................................
2 . User perception of service quality 20
..........................................................................
3 . User perception of governance and corruption 22
4 . Comparison of fve Indian projects on overall client impact
...................................................... 24
.................................. 5 . Variability of client impact
across different delivery centers of a project 28
IMPACT ON AGENCIES
................................................................................................................................
29 IMPACT ON SOCIETY
..............................................................................................................................
34
CHAPTER 4: ASSESSMENT OF MEASUREMENT FRAMEWORK
................................................ 36
...............................................................................................
CONCLUSIONS AND FUTURE ACTION 38
.....................................................................................................................
SUMMARY OF KEY FINDINGS 38 L m r r ~ n o ~ s OF THE STUDY AND AREAS
FOR FURTHER RESEARCH
.......................................................... 40
ANNEX I PRELIMINARY MEASUREMENT FRAMEWORK
...................................................... 42 ANNEX I1
LIST OF ICT PROJECTS COMPLIED FOR WHICH EVALUATION DOCUMENTS WERE
ANALYZED
..........................................................................................................
48 ANNEX I11 SURVEY AMONG TARGETED CLIENTS. AHMEDABAD MUNICIPAL
CORPORATION
..........................................................................................................................................
55 ANNEX IV PROFILE OF RESPONDENTS
..........................................................................................
61 ANNEX V IMPACT ON CLLENTS OF PROJECTS IN INDIA
........................................................ 62 ANNEX
VI FACT SHEETS ON PROJECTS STUDIED
......................................................................
63
........ ANNEX VII RATING OF CLIENT IMPACT (AFTER ELIMINATION OF
SCALE BIAS) 102 ANNEX VIII LOCATIONWISE ASSESSMENT ON DIFFERENT
DIMENSIONS ......................... 103 ANNEX IX A METHODOLOGY FOR
POSTEVALUATION .........................................
........... 105
vii
-
. ............... TABLE I TYPES OF ICT APPLICATIONS DEPLOYED IN
THE PUBLIC SECTOR 6 TABLE I1 . KEY OUTCOME DIMENSIONS
......................................................................................
13 TABLE III . LIST OF SEVEN PROJECTS FOR DETAILED STUDY
.............................................. 15
. ................... TABLE IV WAGE LOSS (RS) FOR MANUAL AND
COMPUTERIZED SYSTEMS 20 TABLE V . PERCEPTION OF COSTITIME
REDUCTION: TAX ONLINE AND CHILECOMPRA
.........................................................................................................................................
20 TABLE VI . AMOUNT PAID TO FUNCTIONARIES AND
INTERMEDIARIESIAGENTS (RS).23
. ............... TABLE VII DESCENDING ORDER OF IMPROVEMENT IN
COMPOSITE SCORES 25 . TABLE VIII TOP FOUR DESIRED ATTRIBUTIONS OF
SERVICES ............................................. 25
TABLE M . KEY INDICATORS FOR ASSESSING IMPACT ON CLIENT
.................................... 28 TABLE X . IMPACT ON AGENCY
......................................................................................................
30 TABLE XI . SAVINGS IN COST TO CUSTOMERS-ESTIMATES FOR ENTIRE
CLIENT POPTJLATION
..............................................................................................................................................
34
............................ . TABLE XI1 ATTITUDE TOWARD
E-GOVERNMENT-INDIAN PROJECTS 35 TABLE XD[I . DETERMINATION OF
SAMPLE SIZE
........................................................................
107
.................................. . FIGURE I E-GOVERNMENT
PROJECT SUCCESS APPRAISAL MODEL 9 FIGURE I1 . NUMBER OF TRIPS SAVED
............................................................................................
18 FIGURE 111 . TRAVEL COST SAVED(RS)
............................................................................................
19 FIGURE IV . WAITING TIME SAVED (MINUTES)
...........................................................................
19 FIGURE V . DIFFERENCE IN OVERALL SERVICE QUALITY SCORE
...................................... 21 FIGURE VI . ERROR RATE
(PERCENTAGE)
......................................................................................
21 FIGURE VII . PROPORTION PAYING BRIBES (PERCENTAGE)
..................................................... 22 FIGURE
VIII.DIFFERENCE IN OVERALL GOVERNANCE SCORE
........................................ 23 FIGURE M . RADIAL
CHARTS REPRESENTING CLIENT IMPACT
............................................ 26
..................................... . FIGURE X INVESTMENTS AND
OPERATING COSTS OF PROJECTS 31 FIGURE XI . PERCEPTION OF
SUPERVISORS OF IMPACT ON AGENCY (PERCENTAGE) . 33 FIGURE XI1 .
PREFERENCE FOR COMPUTERIZATION (PERCENTAGE)
.................................. 38 F1GU.R E XIII.INTERACTION
TREE-* REFERENCE FOR COMPUTERIZEDMANUAL SYSTEM BASED ON USER
PROFILE
....................................................................................................
69
viii
-
An increasing number of governments are investing in information
and communications technology (ICT) as an integral part of their
strategies to improve public sector administration and services.
The efforts to automate back-office functions and to introduce
management information systems started decades ago. With the advent
of the Internet and rapid development in technologies, ICT has made
its way to front-end service delivery through such means as a
government portal. While ICT has become more common in contemporary
governments, evidence to date on the outcomes of ICT investments in
the public sector has not been clearly established. This is
particularly valid for ICT investments in developing countries. On
one hand, a number of success stories in developing countries are
widely lauded as best practices, but on the other hand, a reported
high rate of failure based on expert opinions and case studies
suggests high risks involved.
This study was launched to make a modest beginning for
systematic assessmenit of ICT investments in the public sector in
developing countries. The effort was to go beyond the anecdotal
assessments that were commonly available. The objectives of the
study include: (i) to develop and present a framework to conduct
analysis on the innpact of public sector investments in ICT
applications; (ii) to assess the effect of ICT investments on
performance measures for a selected set of projects and identify
the key determinants of economic, organizational, and social
impact, based on analysis of sample cases; and (iii) to propose
guidelines and a methodology that can be used by subsequent studies
to work with more comprehensive data sets and expanded metrics of
outcome to assess impact of future projects.
Existing literature on impact assessment methodology was
reviewed to clefine a measurement framework. The proposed framework
focuses on the idea of measuring the total value delivered by a
project to different stakeholders. It recognizes that somle part of
the value for each stakeholder can be monetized and other parts
need to be ;assessed qualitatively. The aim is to measure
performance of the manual and computerizetl system on the following
five dimensions: (a) cost of access to clients, (b) client
perception of quality of service, (c) client perception on quality
of governance, (d) agency cost and revenue, and (e) employee
perceptions about process changes. Difference between the
computerized and the manual system was analyzed for each dimension,
and statistical significance of the difference was evaluated.
The framework was used to make detailed assessments of five
mature wide-scope service delivery projects from three states in
India and two projects from Chile. In the Indian projects, online
operators deliver services at designated service centers. In these
projects, data was collected from 240 randomly selected respondents
from eight stratified locations. The responses encapsulated the
experience of clients with the use of the computerized delivery
system as well as the manual system. The survey covered 120
-
questions grouped under five dimensions on which impact was
being assessed.
In the two Chilean projects, services are accessed through a
portal. In each of these, projects data was collected through an
Internet survey of more than 600 users of the computerized system,
and through focus group discussions with employees of the
implementing agencies and citizens. Data was also collected from
agencies and employees through structured formats and surveys.
Key findings from the study are primarily based on the detailed
impact assessment of five Indian projects where data collection was
comprehensive. Results from the two Chilean projects confirm many
of the findings from the Indian projects, even though the nature of
application and country context were quite different. Although the
sample size was relatively small, the estimates of many different
types of impacts were found to be significant at 99 percent levels.
The study points to the following key features of the evaluated
projects.
Survey respondents who have used both the manual and
computerized systems in the five projects in India indicated an
overwhelming preference for computerized service delivery. Their
preference was supported by specific areas where concrete benefits
have accrued to them. In most cases, the cost of accessing service
for sample users was reduced because the number of trips required
to be made to the concerned offices were reduced significantly and
waiting time came down by nearly 50 percent. The quality of service
delivery and of governance was also perceived to have improved with
computerization.
There is considerable variability in the overall composite
scores across five projects. A composite score that could be
compared across projects was computed from a rating of each project
on a list of 16 to 18 common attributes of an e-government
application. Respondents were also asked to select the three most
important attributes. Applying user-assigned importance to
different attributes, a weighted composite score was calculated for
each project for the computerized as well as the manual system. The
difference in the composite score between the two systems indicated
the degree of perceived impact. The differences in scores for
manual and computerized systems range from 0.15 to 1.6 on a
five-point scale, while a difference of 1.6 points indicates a
significant difference, and 0.15 represents virtually no
improvement. In general, these ratings corroborate well with data
on cost savings to customers, perception of quality of governance,
and quality of service collected through an independent set of
questions. The distinct preference for computerized systems by the
clients seems to suggest that even small gains for the users can
trigger a significant positive change in perception about the
overall quality of service delivery systems.
Although the outcome is mixed in reducing corruption, ICT seems
to have the potential for significant reduction in corruption in
service delivery, as indicated by one project. Less than one
percent of surveyed users of one of the three projects where
significant corruption was reported in the manual system responded
that they paid bribes to officials after the services were
computerized.
-
In the other two projects the impact was marginal, as corruption
continued at a significant level in the computerized systems. In
the two Chilean projects, the perceived impact on reducing
corruption was marginal.
Estimates of incremental costs for agencies (including amortized
investment) per transaction in all the Indian projects are less
than the reduction in direct cost of accessing the service reported
by the clients. This indicates that the idea of a user fee that
covers the incremental costs to make the applications economically
viable may not face political resistance. Any resistance to ithe
levy of a user fee comes from the lack of credible information on
costs and benefits delivered by projects. If projects can become
self-sustaining through rlevenues from user fees, private sector
investment is one area that can be tapped. Three of the five Indian
projects have private partners, and the expansion of the fourth
project in rural areas is being planned through private
partnership.
Significant variability of impact across different delivery
centers in a project was observed. This variability should be a
cause for concern for delivery models in which physical service
centers are created. Part of this variability is explained by poor
infrastructure (power and connectivity), particularly in rural
areas;. Given the large variation in activity levels at different
centers, it is often difficult to match the capacity to the demand
at each of these centers. Portal-based delivery accessed via the
Internet can be a solution; however, unequal access to Internet can
put some users at a disadvantage in such systems.
In all cases the ability to cope with growth in transactions was
perceived to have been enhanced. In some cases, computerization
helped in the griowth of transactions. Computerization was
partially responsible for improved tax collections by some
agencies. While the staff in these agencies did not perceive that
cost had been reduced, they did feel that there was a significant
positive impact on the quality of governance.
Surveyed citizens, irrespective of their rural or urban location
or educational background, strongly supported the idea that more
agencies should be computerized.
More projects need to be evaluated from different contexts
before generalizing the conclusions of the positive impact and
economic viability of electronic service delivery projects. The
selected projects have operated for at least three years or more,
implying successful implementation. All the projects except for one
are serving predominantly urban clients. Projects that serve rural
clients could have a very different cost structure and demand
pattern.
The methodology used in this study has produced estimates of
direct benefits for the five Indian projects and an assessment of
perceived change in costs in the Chilean projects. It may be used
in other contexts for similar retrospective assessments. Key
elements of the proposed methodology are: selecting a project;
constructing a project profile; identifying key dimensions of
impact; defining measurement indicators; design of survey
instrument,
-
including design of questions, sampling plan, and a data
validation scheme; and data analysis.
The results of the retrospective evaluation of the seven
projects carried out in this study highlight the relevance of such
an exercise for government officials preparing a plan for an ICT
project and development institutions such as the World Bank. The
results provide a cue for investigating the reasons why a certain
project may not be delivering on a potential area of service
improvement. They also provide a preliminary benchmark for
comparing other ongoing projects and those that will be implemented
in the future. The approach enables an assessment of project
viability in terms of cost and benefits-both in directly measurable
and qualitative terms. The relevance of the proposed approach for
practical use is demonstrated by the fact that the Government of
India has adopted this framework for carrying out an impact
assessment of a large number of projects implemented at the
national, state, and local levels in India.
-
INTRODUCTION AND CONTEXT
It has become the norm rather than an exception to utilize
hformatilon and communications technologies (ICT) in a broad range
of public sector operations in contemporary governments. The term
ICT encompasses diverse technologies such as computing, networking,
hardware, software, telecommunications infrastructure (landline,
mobile, and broadband fiber networks), and broadcasting devices.
The Internet symbolizes this definition of ICT, as it utilizes many
of these technology components. Investments in ICT are made to
create the underlying infrastructure and also to build applications
that utilize the computing power and connectivity. The rapid
technological development in recent times was accompanied by a
significant decline in the prices of hardware, software, and
related auxiliary services. Deregulation and liberalization of the
telecommunications sector resulted in diversification of
information transmission modes, making the delivery of online
services much easier and, most importantly, cheaper. In recent
years an increasing number of projects have been launched under the
label of "e- government," and several governments have identified
this as a key policy priority.
E-government projects deploy ICT applications in the public
sector for different purposes. This portfolio of applications has
been described as a continuum in terms of "stages of growth" from
e-administration to e-services to e-participation (Ranerup 1999;
Heeks 2001). Many governments started using ICT to improve
efficiency and overall performance through automation of
back-office functions and introduc~tion of management information
systems (MIS). Following the trend in the private sector,
governments then expanded the scope of ICT projects to areas
requiring direct int~eraction with citizens. Informational Web
sites provided the entry point for citizen services. As the
technologies matured, the level of interactivity and integration
increased. Currently, attempts are being made to use ICT to induce
wider public participation in policy-.making processes. Through
such means as electronic voting and virtual discussion forums, the
intersections between ICT and governments are no longer limited to
the traditional public administration arena. ICT is beginning to
demonstrate its effect in the fundamental process of policy making,
government-citizen relationships, and, ultimately, how democracy
functions. Table I below illustrates the typical areas where ICT is
being deployed.
-
Table I. Types of ICT Applications Deployed in the Public
Sector
Back-office automation1 / Accounting SystemsFinancial I
Increased efficiency in public Types
operation support I Management Systems I administration
Examples I Intended Outcomes
I Human Resource Management I Transparency in public
financial
I
I Systems ( management I Facility Control Systems
Data gathering and
Electronic delivery of services 1 .
Sector-specific Management Information Systems
Web sitesltransaction portals Integrated online delivery centers
and rural telecenters Online customs, tax, issuance of
certificates, and municipal services
E-discussion
Improved expenditure control
Effective performance monitoring Improved decision making
Reduced costs for transacting with government for citizens and
business Enhanced revenue, greater transparency, and reduced
corruption
Citizen engagement
While an increasing number of governments are investing in ICT,
evidence to date on the outcomes of ICT investments in the public
sector has not been fully established. This is particularly valid
for ICT investment projects in developing countries, some of which
are financed by development institutions such as the World Bank. On
one hand, the recent success of a few projects in improving the
investment climate, improving delivery of services, and creating a
market for information in rural areas has reinforced the idea of
ICT for development. On the other hand, evidence of failed projects
has drawn attention to the level of risk involved in
implementation. A failure rate of more than 50 percent based on the
opinions of 15 experts and student submissions is widely cited1 in
this context. A recent study commissioned by the Department for
International Development (DFID) is researching firm-level impact
of ICT in developing c~untr ies ,~ but similar studies have not
been undertaken for the public sector. A report by the United
Nations (2003) laments the fact that documented research on the
social or economic impact of e-government development is virtually
n~nexistent.~ While multilateral organizations such as the World
Bank fund nearly seven billion dollars of ICT investment as part of
their lending programs and grant assistance to client countries,
evidence to date has been ad hoc, highlighting the need for
systematic assessment of the impact of these applications.4
' See Heeks 2003. "Success and Failure Rates of E-government in
Developing/Transitional Countries: Overview." IDPM, University of
Manchester, U.K., e-government for development Web site:
http:llwww.e-devexchange.orgleGov1sfoverview.htm
An ongoing study sponsored by DFID at the London Business School
that is collecting data from firms in India and Brazil to establish
a relationship between ICT investments, profitability, growth and
productivity at the firm level. Degree of collateral organizational
changes will be studied as a mediating variable.
United Nations 2003. "World Public Sector Report 2003:
E-Government at the Crossroads," p. 61. New York.
The report quotes: "On the World Bank side, the largest, yet the
least monitored, investments in ICT are IT components of projects
in other sectors." See World Bank Group. April 2002. "Information
and Communication Technologies: A World Bank Group Strategy," p.
36. Washington, D.C.
-
This study aimed to ascertain the extent to which the intended
outcomes from ICT applications in the public service delivery have
been achieved. It is an attempt to make a modest beginning to go
beyond the anecdotal evidence currently available on the impact of
ICT investments in the public sector in developing countries. More
specifically, it aims to:
Develop and present an analytical framework to conduct analysis
on the innpact of public sector investments in ICT applications.
Assess the effect of ICT investments on performance measures for a
selected set of projects and identify the key determinants of
economic, organizational, and social impact, based on analysis of
sample cases. Propose guidelines and a methodology that can be used
by subsequent studies to work with more comprehensive data sets and
expanded metrics of outcome to assess impact of future
projects.
To this end, it assessed the extent to which a number of sample
projects have resulted in direct and indrect value for: (i) clients
(citizens and businesses), (ii) the agencies that implement
systems, and (iii) the broader society. Based on the findings, this
report presents a systematic framework and methodology to analyze
the impact of future public sector investments in ICT
applications.
The study focuses on the use of ICT in the delivery of public
services. The j.ntended outcomes from the four broad types of
applications indicated in Table I are important components of
governance reform leading to the attainment of key social and
economic development goals. The focus on the service delivery
aspect was chosen because ithis area directly affects the nature of
interactions between governments and citizens. It is, therefore,
the area where the change in the quality of governance is most
visible to citizens and businesses.
This report is organized under six major headings. The
Introduction discusses the scope and objectives of the study. In
Chapter 1, the review of literature on existing frameworks and
methodologies for impact assessment of e-government projects is
presented. Chapter 2 describes the evaluation framework and
methodology as applied to the assessment of seven projects from
India and Chile. The types of analyses that were carried out on the
survey data collected from users and agency staff for the projiects
are also presented. Results from the analysis are presented in
Chapter 3. Chapter 4L briefly discusses the assessment framework
used in this report. Finally, Conclusions ant1 Future Actions
summarizes the key findings and discusses the usefulness of the
study for developing countries.
-
CHAPTER 1: ASSESSMENT OF PUBLIC SECTOR ICT PROJECTS
Many ICT projects have been launched in different countries that
were initially dubbed "success stories," but which eventually
failed to make any significant impact. For example, a recent EU
report (2003) based on a study of over 100 projects found that
success with realizing concrete and measurable economic benefits as
a result of e-government initiatives remains minimal. In the
context of developing countries, many projects launched have not
led to any significant improvement in the delivery of government
services (Benjamin 200 1, Heeks 2002, Kanungo 2003).
While a significant amount of academic and policy studies have
been conducted, these have so far proved inadequate in providing
policy makers with a systematic framework for evaluation. One group
of studies provides macro-level estimations of e- government
activity using appraisal indices focusing on supply-side,
quantifiable measures such as Web presence of government, network
coverage, institutional and regulatory support, and human capital
provision.5 Such factor-based assessments of e- readiness do not
have immediately obvious and tractable policy implications. They
tend to focus almost exclusively on measurement of physical access
to certain types of ICT without incorporating issues such as
affordability, appropriateness, ICT capacity and training, and the
regulatory and macroeconomic en~ironment.~ The second group of
studies, which have occurred in a largely anecdotal, piecemeal
fashion, provide project- level evaluations with little prospect
for synthesis from past approaches. A small number of studies have
focused on the impact of projects in terms of how they affect both
citizens and government agencies, but findings have seldom been
linked to broader national, regional, or global trends; for
example, in terms of changes in the approach to development and
governance policy (Madon and Kiran 2002, Grant 2005, Heeks
2003).
Recent research in information systems, on the other hand,
focuses on understanding underlying processes of change as a
necessary input toward impact assessment. Of importance here is the
evolving historical and institutional background within which an
ICT project is implemented. Such organizational background has
shown to influence the process by which resources allocated for an
ICT project leads to a successful or unsuccessful system (Avgerou
and Madon 2004).
The focal unit of analysis in the growing body of literature on
enterprise information systems is the project in which the
information systems success model presented by Delone and McLean
(1992) is widely cited. The model is based on an analysis of 180
papers and proposes six major variables of impact which are
interlinked with each other-systems quality, information quality,
information systems use, user satisfaction,
Examples of such frameworks are (1) UNPAN (2004) E-Government
Readiness Report, www.unpan.org and (2) Brown University (2004)
Global E-Government Report, available on
www.insidepolitics.orpr/e~ovtO5int.pdf
www.bridges.org (2005, May). "E-ready for What? E-readiness in
Developing Countries: Current Status and Prospects Toward the
Millennium Development Goals." http://www.infodev.orpr/fiIes/2O49
file InfoDev E Rdnss Rpt revllMay05.pdf
-
individual impact, and organizational impact. Impact on the
individual was later broken down into economic benefit and social
benefit (Seddon and Kiew 1996).
The enterprise information systems success model provides a
generic approach to analyzing informatization in organizations in
terms of the system, usage of th.e system, and its effect. The goal
of a public sector organization is, however, more complicated than
an enterprise information system, as its function includes both
internal management and public service delivery. While enterprises
view economic benefits as their strategic goal, the government
needs to take both economic and social benefits into account. The
development of a framework for appraising the impact of ICT
applications in Ithe public sector, therefore, depends on the
formation and sustenance of a supporting institutional,
infrastructural, and policy environment. The e-Government project
Success Appraisal Model (Figure I) incorporates a variety of
objectives being pursued by governments. It aims to serve as a
basic reference framework and important building block for
evaluating e-government projects. Rather than impose a rigid
template for conducting the appraisal exercise, the model is
intended to be a guide to improve the impact of systems,
acknowledging that different countries may have different
contextual conditions and
- -
motives for implementing e-government applications.
Figure I. E-Government Project Success Appraisal Model
Foundation and environment of e-government I Source: Hu, Y ,
Xiao, J. Pang, J. and Xie, K. (2005). A Research on the Appraisal
Framework of e- Government Project Success. Proceedings of the 7th
International Conference on Electronic C~xnmerce.
These discussions have been incorporated in assessment
frameworks for IC?' projects developed by different public sector
agencies and The U.S. govt:mment7s Performance Reference Model
(PRM), for example, is a widely cited model that aims to
' The indicative items are based on a review of the following
documents: i) Performance Reference Model of the U.S. Federal
enterprise architecture framework used by the Office of Management
and Budget in U.S. federal government; ii) European Commission,
eGovernment Economics Project (eGEP), Measurement Framework Interim
Version Deliverable D.2.2, June 27, 2005; iii) Rarna Rao, T. P.,
Rao, V. Venkata, Bhatnagar, S. C., and Satyanarayana, J. (2004).
"E-Governance Assessment Fraimeworks." Available on
http://egov.mit.gov.in; iv) Lau, E. (2005). Electronic Government
and the Drive for Growth and Equity. OECD; and v) Lanvin, Bruno
(2005, November). "METER E-Strategies Monitoring and Evaluation
Toolkit." E-Development: From Excitement to Effectiveness, ed.
Robert Schware, prepared for the World Summit on the Information
Society, Tunis, Tunisia. World Bank.
-
measure value creation in the public sector. The PRM is a tool
to evaluate the impact of IT investment across federal government
agencies and serves as the overall driver for a set of interrelated
reference models developed under the U.S. Federal Enterprise
Architecture Program. The model is used to facilitate cross-agency
identification and analysis of duplicative investment, gaps, and
opportunities for collaboration and integration across agencies,
drawing on the best of existing approaches to performance
measurement in the public and private sectors and harnessing
various tools and methodologies frequently used in the private ~ e
c t o r . ~ The framework builds from value chain and program logic
models to reflect how value is created as inputs (such as
technology) are used to create outputs (through processes and
activities), which in turn impact outcomes (such as mission,
business, and customer results). Guiding the entire PRM framework
are strategic outcomes representing broad policy priorities that
drive the direction of government.
The national e-government frameworks developed by the Australian
(Commonwealth of Australia 2004) and German (Federal Ministry of
the Interior 2006) governments focus broadly on the same dimensions
as the PRM but with an emphasis on overall agency impact. These
methodologies offer two levels of impact assessment: first, in
terms of how the project provides a business case justification of
expenditure and policy and whether it meets the targets set for it,
and second, in terms of how the project meets the goals of the
agency concerned and, in turn, how this helps in meeting wider
government strategies.
As many e-government initiatives worldwide are struggling to
demonstrate the expected benefits, there is a growing demand to
understand the 'processes' that deliver a return on investment and
generate value (EU 2003). Common best practice factors for value
creation in e-government projects have been well-documented in
several reports by governments and consultant firms (Accenture
2002, Gartner 2002, EU 2003). An example is the study conducted
jointly by the Danish Ministry of Finance and Accenture that
identifies two primary dimensions of value: (i) value to investors
in terms of tangible financial benefits, cost savings, cost
avoidance, and increased revenue; and (ii) value to users in terms
of improved services, reduced cost and time savings to citizens,
and reduced administrative burden to businesses. Investment in
government employees and technological infrastructure are
identified as key enablers for creating value through e-government.
The focus has been on using business case methodologies from the
private sector to demonstrate the economic value of e-government
projects.
The calculation of time and money spent in finding and using
public information are the most direct and measurable benefit of
e-government applications to users, as shown in recent Organization
for Economic Co-operation and Development (OECD) and European Union
(EU) studies (Lau 2005, eGEP 2005). These reports identify improved
revenue collection, lower costs caused by efficient processing of
transactions, and a reduction of administrative burdens caused by
simplification or elimination of procedures as some direct impacts
that can be measured. Financial savings to users in terms of time
and money spent in finding and using public information are the
most direct and measurable
Some of the main techniques included in the PRh4 are strategic
planning and analysis, business cases, value chain, applied
information economics, and the balanced scorecard.
10
-
benefits of e-government applications for clients. A significant
amount of work has also been done on studying users' perceptions of
quality in terms of attributes such as accessibility, attitude of
staff, cost of service, provision of information, .procedural
fairness, and convenience.
While the benefits of process improvements in the public sector
are acknovvledged in many studies, the links to increased
productivity of government services are not straightforward. The
expected links are not easily inferred through the expc:riences of
ICT implementation in the business sector. The distinction between
measuring productivity increases in the manufacturing and services
sectors has been articulated in a recent McKinsey report in which
the authors document how service businesses around the world are
trying to boost their productivity (Harmon et al. 2006). Unlike
consumer goods, which are relatively similar and therefore
comparable, the report describes how there is often great
divergence in the way services are recorded and monitored among
groups even in the same organization. Service companies, therefore,
need to develop internal benchmarks to find their own overall best
practices and to see where and how they can be achieved. These
internal benchmarks may differ from company to company highlighting
the importance of context in impact assessment studies. The same
would apply for measuring improvement in government services.
Calculation of net economic benefit of an IT project has proven
to be challenging, as outcomes are multidimensional and composed of
both quantitative and qualitative indicators. Further attention is
required to be given to the linkages among issues of quality of
service, governance, wider impacts on society, and ICT investment.
Newer frameworks are evolving around a notion that an exclusive
focus on financial costs and benefits for the government ignores
many important noneconomic benefikg One such benefit relates to
improvements in various aspects of governance activity. Another
noneconomic benefit relates to the addressing of wider policy
priorities, which, in the context of developing countries, could
constitute the priorities as articulated in the United Nations
Development Programme (UNDP) Millennium Development Goals.
This study builds on these recent studies on the assessment of
ICT projects in the public sector. These studies indicate the value
of ICT to be its contribut:ing to the achievement of organizational
priorities. They also call for an assessment framework applicable
for different ICT projects, so that systematic evaluation of the
role of ICT can be undertaken. Following these discussions, this
study focuses on understanding and documenting how inputs lead to
outcomes in particular projects, given their contextual conditions.
The assessment framework focuses on the idea of measuring the total
value delivered by a project to different stakeholders and takes a
balanced approach between case-study and quantitative analysis. It
recognizes that some part of the value for each stakeholder can be
monetized and other parts needs to be assessed qualitatively.. The
next section provides a description of the measurement framework
and the assessment methodology of this research.
Lau, Edwin (2005). Electronic Government and the Drive for
Growth and Equity. OECD.
11
-
CHAPTER 2: MEASUREMENT FRAMEWORK AND DATA ANALYSIS
MEASUREMENT FRAMEWORK AND METHODOLOGY
Most of the assessment models proposed in the literature have
not been used in the context of a developing country and many have
not yet been applied in practice even in developed countries. Nor
do they account for a variety of delivery models used in developing
countries such as common service centers and franchised outlets
that can retail e-services offered by the government. Practical
issues of paucity of data have not been taken into account,
particularly in case of a developing country where baseline surveys
are not done and monitoring and evaluation systems are weak.
Recognizing that adequate data for quantitative assessment is not
available for most of the projects, collection of such data using a
standard measurement framework for a sufficiently large number of
projects is the first task that needs to be undertaken. Since
impact assessment is the key objective, establishing
counterfactuals is an important element of the proposed measurement
framework.
A number of empirical studies suggest that ICT has had an impact
in improving the performance of private sector organizations,
particularly in developed countries. However, regarding ICT
investments by the public sector in developing countries, many
researchers have noted that past evaluation studies have not used a
common framework or methodology and that rates of success or
failure have been declared based on purposive samples.'0 Since ICT
is introduced mostly in the context of governance reform to improve
efficiency, effectiveness, and transparency of governments, a
crucial first stage is to ascertain to what extent these intended
outcomes from ICT applications in the public sector have been
achieved.
A common measurement framework evolved on the basis of existing
frameworks has been used in this study. An e-service delivery
project impacts three groups of stakeholders: (i) clients receiving
the service; (ii) agency (including several partners) that delivers
the service; and (iii) the larger society consisting of citizens,
businesses, government as a whole, and civil society. The impact
can be assessed in terms of a variety of outcomes experienced by
each type of stakeholder. Table I1 lists key dimensions of outcomes
for each type of stakeholder. The complete assessment framework is
provided in Annex I.
10 Peters, Rob M., Janssen, Marijn, and van Engers, Tom M.
(2004, August). "Measuring E-Government Impact: Existing Practices
and Shortcomings." Proceedings of the 7" International Conference
on Electronic Commerce.
-
Table 11. Key Outcome Dimensions
Client -
Agency (including partners in
Stakeholders Economic (direct and indirect) Governance
(corruption, accountability, transparency, participation) Quality
of service (decency, fairness, convenience, etc.) I Economic
Governance (direct (corruption, and indirect) accountability,
transparency, participation)
Key Dimension of Impact
implementation) I Performance on key noneconomic objectives
~ o v e i m e n t as a whole responsiveness) Civil society
Development goals
Society
I ~t t i tude toward computerization of public service
delivery
Process improvements
Governance (corruption, accountability, transparency,
participation,
The generators of these outcomes are inputs and business
processes. 'Therefore, assessment needs to incorporate project
information such as inputs (technology, human capital, etc.);
process outcomes (reengineered processes, shortened cycle time,
improved access to data and analysis, flexibility in reports); and
organizational processes (institutional arrangements,
organizational structure, and other reform initiatives of the
government that might have influenced the outcome of the ICT
project)."
As a first step to apply the framework to actual projects, a
list of projects from developing countries intended for public
sector operations that have some level of impact evaluation
conducted and documented was compiled. Projects included in the
list had a comprehensive scope in terms of functions that had been
automated in an agency or were projects that had a statewide or
region-wide coverage in large countries and a national coverage in
small countries. The investment in such projects was generally
upward of $2 million. Thus, pilot projects and projects with a
small scale and scope were excluded. Different types of
documentation were collected on these projects, especially
tlocuments that were evaluative. The list of about 75 such projects
is presented in Annex 11. Some observations were made based on this
sample:
Evaluative studies had been done to serve a variety of purposes.
Some studies looked at implementation success in terms of whether
the systems were functioning as they were designed to, or the
degree to which the specified outcomes were achieved. Some studies
looked at long-term sustainability and scope for replication; l2
some measured the benefits that were delivered to agencies.13 A few
focused on benefits to the clients.I4 Hardly any comprehensive
study assessed the impact on all the stakeholders and covered both
shont-term and long-term direct and indirect impacts. A few studies
had carried out a cost-benefit analysis. Often, evaluation studies
had been done by agencies that wer~e likely to
l 1 Hu, Yong, Xiao, Jing Hua, Pang, Jia Feng, and Xie, Kang
(2005, August). "A Research on the Appraisal Framework of
E-Government Project Success." Proceedings of the 7th International
Conference on Electronic Commerce. 12 Government of India, Ministry
of IT, has commissioned quick assessment of 29 projects. l3 Korea's
eprocurement agency has evaluated the impact on dfferent government
agencies using the system. l4 Global Knowledge Sharing Program had
four Indian projects evaluated where clients were surveyed.
-
have an interest in showing a positive outcome. A variety of
approaches had been used for evaluation. These included surveys,
expert opinion, ethnographic studies, and internal assessments
produced by - lending agencies. Different studies of the same
project showed very different outcomes, thus indicating a lack of
credibility of results.15 Part of the reason for different outcomes
was the use of very small samples and a lack of rigor in sampling
and collecting data from clients of the systems. The results could
therefore not be easily generalized over the entire population of
clients. The studies evaluated the functioning of the computerized
system but were not able to assess the difference made by ICT use,
as the need for counterfactuals (evaluation of systems as they
worked before computerization) was ignored. Often, the impact of
ICT use was not separated from other interventions that were made
simultaneously with the computerization effort. Since different
studies did not use a standard methodology, it was difficult to
compare the outcome of one project with that of others.
From the compiled list of 75 projects, a sample of seven mature,
wide-scope and -scale projects were selected for further study
(Table 111). The list includes Bhoomi, Karnataka valuation and
e-registration (KAVERI), Computer-aided administration of
Registration Department (CARD), eSeva, and Civic centers of
Ahmedabad Municipal Corporation (AMC) projects from India; and Tax
Online and ChileCompra projects from Chile. They were chosen also
to represent the countries in different stages of overall IT
readiness and evolution.
l5 For example, the Bhoomi project of issuing copies of land
titles has been evaluated by the Public Affairs Council, reporting
significant positive outcomes, including reduction in bribes.
Recent studies by a team from the Massachusetts Institute of
Technology and the Indian Institute of Information Technology
Bangalore found that corruption had not declined and major benefits
were derived by land sharks.
-
Table III. List of Seven Projects for Detailed Study
r (RTC)'~ and filing of a
administration of
eSeva
, state, and local governments and public utilities and are now
used
Corporation (AMC) I property tax, issue of birth and death
certificates, and issue of shop licenses. Chile Tax Online Online
tax filing was implemented by the Servicio de Impuestos Internos
(SII),
which launched its portal in 1998. Attained maturity by May 2002
when 55 percent of Chilean citizens were filing their tax
declarations online. By 2004, half
I of all tax returns were prepared automatically and 20 percent
partially. In 2006,
I platform in 2003. Provides for online publication of
procurement notices and e- ChileCompra
I mail alert services for registered businesses. Over 90 percent
of private- and
97.4 percent of all declarations were conducted online.
Portal was launched in 2000, but came into full form as an
e-procurement
I ~ublic-sector organizations have availed themselves of the
Drocurement svstem.
For each project, the measurement framework was converted into a
set of data collection instruments, including: i) a profile of the
project identifying services, clients, and other stakeholders; ii)
agency-level data on activity levels, investments, and operating
costs; iii) a client survey questionnaire covering direct cost of
access, quality of seirvice and governance; overall satisfaction,
and perception of ICT projects based on exposure to different ICT
applications; and iv) an employee survey for perceived impact on
work, efficiency, and effectiveness. Measurement instruments were
translated into local languages. A sample client questionnaire for
AMC civic centers is shown in Annex III.
The survey conducted in India tried to assess both the manual
system and the computerized system that replaced it on all the
indicators to the extent possible. An analysis of the differences
between the old and the new system was done to provide a measure of
impact. A sampling scheme for survey of clients through
structured
l6 Besides its other uses, this document is required for
availing crop loans from banks.
-
CHAPTER 3: ASSESSMENT RESULTS FOR INDIAN AND CHILEAN
PROJECTS
This section presents a comparative analysis of the Indian and
Chilean projects from the perspective of the clients. Impacts for
the key dimensions of cost, quality, governance, and overall
satisfaction are analyzed. A profile of each project, detailed
analysis of impact on clients and agency, and perception of
employees is presented as a case study in Annex VI.
A COMPARATIVE ANALYSIS OF PROJECTS FROM CLIENT PERSPECTIVE
1. Cost to users
The use of ICT in public service delivery in India has reduced
the number of trips surveyed users have to make to complete a
transaction (Figure 11). The survey shows that in all five
projects, there were savings in the number of trips citizens made
to service facilities where public services are delivered (see
Annex V for more detail). l8
Figure 11. Number of Trips Saved Greater formalization of
processes after process reform, quicker retrieval of data from
computerized databases, and automation of tasks such as writing and
copying documents, as compared with manual methods, has reduced the
total processing time of a service request in all applications.19
This has resulted in increased predictability in service delivery
and reduced the number of trips.
Along with reduction in the number of trips, the travel costs
per transaction have generally declined (Figure 111). The only
exception is the Bhoomi project in which travel costs for obtaining
RTC
l8 A paired t-test was applied to assess whether the differences
between the means of the number of trips for the computerized and
manual systems were significantly different. For details of the
test statistic, refer to Annex V and project fact sheets in Annex
VI. l9 In the case of AMC, the time elapsed has been reduced from
9.8 days to 5.3 days for certain types of transactions, while in
CARD, the total time required for registrations has come down from
4.69 to 1.82 days. For details, refer to project fact sheets in
Annex VI.
-
have increased by Rs 79.96 per transaction and for mutations by
Rs 81.38. This is primarily caused by the change in the location of
service offices. Under the manual system the RTC was issued in a
village, but with computerization users need to travel longer to
the higher-level government office (taluka headquarters) to receive
the same service (more detail in Annex V). The opportunity cost for
users, which is represented by wage loss because of travel, is,
therefore, larger.
Figure 111. Travel Cost Saved(Rs) , The increase in travel costs
for
Bhoom'- Bhoond- KAVERI CARD eSeva AMC RTC Mutation
expertise to support elaborate ICT applications. To planning to
expand access in rural areas by allowing I database for issue of
RTCs.
Waiting time at the service centers was reduced in all the
projects. 20 Among those projects in which the reduction in waiting
time were statistically significant 21 , the savings ranged from
16.16 minutes (AMC) to 96.24 minutes (CARD), amounting to a nearly
50 percent reduction from the waiting time in the manual system
(Figure IV). Reduction in waiting time would have direct impact
on
users of the Bhoomi project demonstrates one of the key
challenges of service delivery using ICT. The change in the point
of service delivery frlom the village office to the taluka
headquarters was necessitated to ensure the availability of robust
information infrastructure and technical support to operate the
system. Many parts; of developing countries, partilcularly those
outside major cities, may lack infrastructure and technical
address this problem, Bhoomi is ~rivately owned kiosks to access
its
Figure IV. Waiting Time Saved (minut
e
s)
Bhoom- Bhoomi- KAVWl CARD eSeva A M : RTC Mutation
20 A paired t-test was applied to assess whether the differences
between the means of the waiting time saved for the computerized
and manual systems were significantly different. Refer to Annex V
and fact sheets in Annex VI for details of the test statistic. 2 1
Only Bhoomi RTC was not statistically significant. See Annex V for
more detail.
-
user costs through drop in foregone wages, as indicated in Table
IV.
Table IV. Wage Loss (Rs) for Manual and Computerized Systems
N=sample size; S.E=standard error
While the difference between manual and computerized systems on
all the cost indicators are statistically significant (in most
cases at a confidence level of 99 percent), the accuracy of the
estimate of the difference in direct costs can be improved by using
larger samples.
In the two Chilean projects actual changes in cost of access
were not measured directly. Instead, the impact on costs and time
reduction was measured as a perception on a five- point scale.
While citizen and business users of the Tax Online system reported
a - significant perceptional reduction in the cost and time, in
ChileCompra the perceived improvements were relatively marginal
with a higher proportion of respondents perceiving no change.
Table V. Perception of Costrrime Reduction: Tax Online and
ChileCompra
Costs required to search for business 3.28 1 Change in the
economic cost to process 4.02 opportunities (0.07) taxes (0.07)
Costs required to participate in 3.23
hid din^ (0.07) I - . .. - . . . I \ - . - ' I
Time Reduction Time required to search for business
Any change in the total amount of work 3.40
4.47 opportunities time and effort required to process
(0.07) (0.05) Time required to participate in taxes? 3.44
bidding (0.08) Note: Standard error in ( ).
2. User perception of service quality
Users rated the quality of services delivered using computerized
systems as better than that by manual systems. For all five
projects, users rated the computerized
22 services higher in overall quality (Figure V). The
differences were statistically
22 A paired t-test was applied to assess whether the differences
between the overall service quality scores for the computerized and
manual systems were significantly different. For details of the
test statistic, refer to Annex V and project fact sheets in Annex
VI.
-
significant. The rate of error, which is an important measure of
quality, was also1 lower for all the projects, except the issue of
RTCs (Figure VI). Bhoomi also shows very significant improvement in
service quality, among the highest in the five Indian projects
assessed in the study.
Figure V. Difference in Overall Service Quality Score
(five-point scale)
Bhoom- Bhoomi- KAVERl CARD eSeva AMC RTC Mutation
Figure VI. Error Rate (percentage)
RTC Mtation
Overall improvement in service quality was reported by users in
the two projects in Chile as well. Both projects received favorable
assessment for their user-friendliness and convenience of services.
The average ratings on the quality of service of the tax online
system were high, with many items being rated around 4.5 in
five-point scale. On an average, users of the tax online system can
complete tax filing in 30 minutes. Abolut one- third of business
users and 3.6 percent of citizen users, however, had to make an
appeal to SII regarding a tax decision. The system was also rated
poorly on responsiveiness to feedback. For suppliers using the
ChileCompra portal, the interviews revealed a marked increase in
efficiency and effectiveness. While earlier a supplier had to
constantly refer to newspapers for knowledge of purchase intent
declared by the government, the ChileCompra portal provides such
information at a single place. In terms of complaint
-
handling, about half of public sector users surveyed had to
appeal to a ChileCompra decision and around one-third of user
organizations had noticed errors in ChileCompra documentation.
Focus group discussions and interviews revealed that many providers
were disgruntled with procurement decisions made through
ChileCompra. Thus, transparency of information appears to have led
to increased complaints from suppliers.
3. User perception of governance and corruption
The proportion of surveyed users paying bribes has generally
declined. The decline is higher in some projects than in others
(Figure VII). For example, in the Bhoomi project, while about 30
percent of users were paying bribes in the manual system, less than
one percent are continuing with the practice in the computerized
system.
("/I 35.0
30.0
25.0
20.0
15.0
10.0
5.0
0.0 Bhoom- Bhoomi- KAVERI CARD eSeva AMC
RTC Mutation
a Manual H Cunputerized
Figure VII. Proportion Paying Bribes (Percentage) The proportion
of transactions in which a bribe was paid in KAVERI continues to be
high at 21.61 percent in the computerized system. A detailed study
of one of the centers indicated that any type of system breakdown
leads to corruption. The breakdown can be on account of an overload
of demand in comparison to the capacity of the system to process
registrations. Agents play a key role in
promoting corruption. Private operators also exhibit
rent-seeking behavior if given an opportunity. Systematizing queues
by appointments helps prevent breakdown.
An interesting phenomenon is that the amount of bribe users pay
is generally higher in computerized systems in all the projects
(Table VI). General inflation in the price levels is perhaps part
of the reason for this. It may also suggest that the complexity of
the types of favor requested through bribes have increased. The
estimates of bribe amounts are less accurate than other results, as
the sample of respondents paying bribes was small (50-75),
resulting in high standard error of the estimates.
-
Table VI. Amount Paid to Functionaries and IntermediariesIAgents
(IRs)
I
Other amount paid to intermediaries/agents (Rs) Manual 39.44
46.13 91.92 256.41 77.27 N 11
Computerized 1 78.75 1 50.00 1 200.75 1 398.61 1 - 1 60.00
N=sample size; S.E.=standard error
The overall governance rating is higher for computerized systems
than ~nanual systems (Figure VIII).23 In the case of Bhoomi, there
was a marked improvement in transparency and fairness, whereas
KAVERI showed hardly any improvement in the quality of
governance.
Figure VIII. Difference in Overall Governance Score (five-point
scale)
Bhooni KAVEU CARD eSeva
23 A paired t-test was applied to assess whether the differences
between the overall governance scores for the computerized and
manual systems were significantly different. These were found to be
significantly different at 99 percent confidence level. For details
of the test statistic, refer to Annex V and project fact sheets in
Annex VI.
-
The results for the projects in Chile appear to complement these
findings in the projects in India. There was a general consensus
among respondents that ChileCompra has resulted in the public
sector being held more accountable for its spending. Focus groups
and interviews revealed that it is possible for a supplier to
verify and challenge why the government had rejected a bid in
ChileCompra; however, ChileCompra was rated quite poorly on all the
governance indicators by the vendors, whereas the public sector
agencies rated it as moderately positive. Public sector users
responded that ChileCompra had somewhat reduced corruption while
there was no such endorsement from private sector users. The survey
recorded strong agreement from public sector players that the rules
and procedures of ChileCompra were clearly stated, while only 26
percent of the respondents agreed with this from the private
sector. Public sector players strongly agreed that ChileCompra
tends to add transparency to public sector operations, while weaker
endorsement was shown from the private sector. The Tax Online
system appears to have improved the perception of fairness among
businesses, as revealed through focus group interviews. All
businesses that have a permit to trade issued from the municipality
are now automatically included in the system if their annual
earning is above a certain threshold. Earlier, many such
enterprises were outside the system. This measure is perceived as
fairer by the business community. There was no strong endorsement,
however, that the Tax Online had either increased transparency,
eliminated corrupt practices, promoted confidential and secure
data, or that the SII takes into consideration the opinion of
citizens while developing norms and rules.
4. Comparison of five Indian projects on overall client
impact
Table VII compares the composite scores on a five-point scale
based on 18 common attributes encompassing the three dimensions
discussed earlier. The score also factors in the importance of the
attribute for each project. Respondents were asked to select three
attributes considered important by them from a set of 18 attributes
of a service delivery system. A weightage was assigned to each
attribute on the basis of the proportion of respondents who
selected the attribute. Weighted scores were computed for each
attribute as the product of the weight and the difference between
the average scores for the attribute on the manual and computerized
systems. The sum of these differences denotes the improvement in
composite scores after computerization.
The results indicate that Bhoorni shows the maximum improvement,
as its manual version had the lowest score. eSeva also shows a very
significant improvement, as the computerized counters are rated
close to very good in the composite score. KAVERI and CARD indicate
only marginal improvement over the manual system. Ranking of
projects using this data, particularly the composite rating, can
represent the degree of success of the project from the point of
view of clients. Bhoomi and eSeva can be rated as very successful
projects, AMC as moderately successful, and KAVERI and CARD as
projects where there is considerable scope for improvement.
-
Table VII. Descending Order of Improvement in Composite Scoresz4
--
Nlanual Computerized Difference
AMC I 3.367 4.118 0.75 1
Bhoomi
eSeva
KAVERI I 3.345 3.897 0.552
2.857 4.461 1.604
3.388 4.658 1.270
Three kinds of attributes have been selected more often than
others, as shown in Table VIII. They include strong preference for
attributes on i) transactional efficiency, ii) improved governance,
and iii) quality as measured by error rate and convenience. An
important conclusion is that these attributes are different for
different projects. If such an exercise was to be done before
designing an application it would provide useful insights into
benefits that need to be targeted and the kind of process reform
required. It is rarely done in practice either before or after the
project implementation.
CARD
Table VIII. Top Four Desired Attributions of Services
3.775 3.929 0.153
Project BHOOMI
Dwired Attribdtiun b~Seruiccs Error-free I No delay in I Less
waiting time I Fewer visits
KAVERI
CARD
Figure IX presents radial charts for manual and computerized
delivery for each project on the key dimensions included in the
assessment framework. The overall score for manual and computerized
delivery is presented as a circle in the same chart. Since
different types of indicators have been measured differently (some
using a five-point perception rating and others as actual
measurements), a methodology of predetermined "goalposts" was used
to normalize them. It reflects the feasible upper and lower limits
to the rnleasures, adopted by the United Nations Development
Programme (UNDP) in computing the Human Development ~ n d e x . ~ ~
Performance on each dimension is expressed as a value between 0 and
1 by applying the formula:
Dimension index = (actual value - minimum value) /(maximum value
- minimum value)
transaction Less corruption
Less time and effort reauired
24 Differences between the composite scores for the computerized
and manual systems were found to be significantly different at 99
percent confidence level. For details of the test statistic (paired
sample t-test), refer to project fact sheets in Annex VI. 25 Human
Development Report 2006, United Nations.
Less waiting time
Less corruption
Convenient time schedule Greater transparency
eSeva
AMC
transaction Greater transparency
Less waiting time
Fair treatment
Good complaint handling system
Less time and effort required Less time and effort required
Error-free transaction Less corruption
Less waiting time
Fair treatment
-
AMC
rbsence of
Quality of Governanc uality of Service
Score on Indicators for Manual System - - - . Overall Score for
Manual System Score on Indicators for Computerized System __
Overall Score for Computerized System
-
These radial charts allow for a quick visual assessment of the
rating of each project based on five key dimensions of client
impact in its manual and computerized versions. The larger the area
of the pentagon is, the higher the rating of the project is. It is
easy to surmise that eSeva has been highly rated. Bhoomi appears to
have shown the maximum improvement over the manual system. KAVERI
has the smallest pentagon; that is, it is rated the lowest and has
registered marginal improvement through computerization. Similarly,
the composite ratings represented by the concentric circles show
very marginal improvement in KAVERI and significant improvement in
Bhoomi and eSeva. The charts also establish that the overall rating
derived from individual components and composite scores are
consistent.
The six key parameters that have been used to generate the
radial charts are presented in Table IX. Scores obtained by
different projects on these indicators after elimination of scale
bias are represented in Annex VII
Table M. Key Indicators for Assessing Impact on Client
5. Variability of client impact across different delivery
centers of a project
, , ' I d a * ' Cost
~ f f i c i e n c ~ 26
Quality of Service
Quality of Governance
Absence of Corruption
Overall Score
There are visible variations in the survey results in different
locations of service facilities. Earlier analysis of client impact
was based on the total number of respondents, typically surveyed in
seven to eight different locations. Data was analyzed to understand
if there were differences by location of service facility on a few
key dimensions across all projects. The mean for each location is
tested against the total sample mean for statistically significant
differences (Annex VIII). On improvements in composite scores,
which is an overall evaluation, there is significant variation in
KAVERI, AMC, and Bhoomi. The variation in CARD is small, as the
improvement itself is marginal. In eSeva the overall improvement is
significant but the variation between locations is small. It is
interesting to note that eSeva is the only application where the
capacity of the service center is designed to match the activity
levels.
Variation is largest among Bhoomi service centers. In
particular, one service center (Location 4) facility scored the
worst in terms of composite score. Respondents evaluated
Farmefer Number of Trips
Waiting Time (minutes)
Quality of Service (five-point scale)
Quality of Governance (five-point scale)
Proportion Paying Bribes (%)
Composite Score (five-point scale)
26 Since it is difficult to predict the worst scenario in the
case of number of trips that would be required to avail a service,
the highest average number of trips (in the manual system of
registration in Karnataka) has been considered for the purpose of
scale equalization. Similarly, for the Eficiency indicator, the
maximum and minimum of actual averages of the time spent at the
service center during each visit have been used as the lower and
upper goalposts respectively.
,M*m Ydwb 0
14.466
5
5
0
5
> < . . < > I
M W m b Value 3 . 3 7 0 ~ ~
184.786
1
1
100
1
-
a 0.575-point deterioration in overall performance with the
computerized system. The increase in travel costs because of the
change in the location of service facilities from villages to
taluka headquarters probably contributed to this result. Users of
Location 4 experienced the largest increase in travel costs after
computerization. Quite unexpectedly, the number of trips per
transaction was also substantially higher in this service center.
Users waited longer at the computerized service facility than under
the manual system. Moreover, the mutation service provided in this
location is the only case among surveyed facilities of the Bhoomi
project in which the proportion of users paying bribes increased.
There was a breakdown in the service delivery process in this
particular facility. While the precise reasons for deterioration
and the frequency of similar incidences in the entire Bhoomi
project require further studies, the situation in Location 4
highlights the comprehensiveness of efforts required to improve
service quality. Computerized system installation by itself does
not ensure positive transformation of services.
Direct economic impact on agencies was expected in terms of
operating costs, collection of tax revenue, and collection of
transaction fee. There are inherent problems, however, in assessing
impact on costs. For different agencies, comlputerized delivery of
services touches different portions of the overall activity
portfolio. For example, AMC is an organization that delivers
large-scale physical service!; such as cleaning of roads and public
areas, public lighting, and maintenance of urban infrastructure.
Computerized delivery of services impacts less than one percent of
all employees. Often, the prevalent costing systems do not permit
identification of operating costs for only the computerization
project. For agencies such as offices of the subregistrar the
impact is more widespread, as the core service is computerized.
As mentioned in Section 3 of Chapter 2, it was difficult to
collect cost data from the manual operations. By and large, the
major cost component was labor. The case studies reveal that human
resources were not reduced in any project after computeri:zation.
In some projects, workload was reduced and, therefore, additional
time could be dLevoted to other tasks. There could be cost
implications in the future, as the transactions are growing at a
much faster rate as compared with labor, as was seen in the case of
KAVERI. The human resource and other costs incurred by the
government have remained within a narrow range for the manual and
computerized systems. After computerization, the total cost as a
percentage of revenue from fees has declined from 6.48 to 4.23
percemt in two years.
Given these limitations, data on impact on agencies collected is
summarized in Table X. Data shows that there has been a significant
increase in revenues from transaction fees and tax revenue
(wherever applicable) in recent years.
-
Table X. Impact on Agency
Total vroiect investment (Rs million) I 216.35 I 400 1 300 1 600
1 250
Bhoomi RTC Mutation
Annual operating expenses (Rs million) I 159.32 I 111.13 i 64.5
1 168.9 1 19.2 Number of clients served annually (million) I 7.62 I
1.33 1 0.33 1 15.3 1 0.29
KAVEKI CARD eSeva
Tax revenue (Rs million)27
Annual number of transactions (million) I 1 1.97 1 1.03 1 2.47 1
1.03 1 37.02 1 0.7 1
AMC
n.a. 1 19,245.07 1 17,282.68 1 n.a. 1 1,974.26 Tax revenue base
(Rs million)
Growth rate in tax revenue
Transaction volume base (million) 1 6.56 1 0.29 1 1.49 1 0.41 1
10.5 1 0.27
n.a.
n.a.
Growth rate in transaction fee (percentage) 1 22.23 1 87.83
11.59 24.74 1 120.15 1 17.51
Growth rate in volume of transactions (percentage)
Transaction fee (Rs million)
Transaction fee base (Rs million)
n.a.: Not Applicable The years for current and base jigures for
tax revenue, transaction volume and transaction fee are set
differently for each project depending on project starting date and
data availability. Please refer to Annex V for more detail. Growth
rates are annualized
9,033.16
28.67
All of this increase, however, cannot be attributed entirely to
computerization. Based on discussions with agency staff, impact on
transactions, fees, and tax revenues was analyzed qualitatively.
Some transactions, like bill payment in eSeva, are mandatory. The
growth likely represents a shift from traditional manual based
payment counters run by agencies and a natural growth in
electricity and water connection. Other transactions, such as RTC
and Mutation, are not mandatory. In addition to overall growth in
land holdings, farmers may have been encouraged to keep their land
records updated because of Bhoomi. In KAVER129 and CARD3' the
transactions of property registration is mandatory. Part of the
growth in other types of transactions3' may be attributed to
computerization. For tax revenue, a more transparent system of
valuation, rate of growth in property transactions, could explain
growth. In AMC the growth in taxes is influenced by a growing base,
better compliance, and accurate valuation; however, in all cases,
computerization appears to have allowed the agencies to cope with
high growth in the number of transactions.
22.23
179.58
98.34
Another key aspect that determines service levels in each
project is the quality of maintenance of the infrastructure at
various locations. A survey of about 10 front-line
4,765.27
23.95
" KAVER17s figures for tax revenue are revenue from stamp duty
Of this amount, Rs 159.7 million was paid to the private operators,
whereas the government's share was
Rs 2,467.3 million. 29 In 200041, when the system was manual,
0.63 million properties were registered. In 2005-06, 1.02 million
properties were registered, representing an annual growth of 10.27
percent. 'O 1n 1998-99 when the system was manual, about 54,000
properties were registered. The registration jumped eightfold in
1999-2000, one year after computerization.
In the case of KAVERI, requests for non-encumbrance grew at
12.21 percent and copies of registration at 2.25 percent.
87.83
36.12
10.24
n.a.
n.a.
651.36
44.72
10.58 16.78 87.74
203.59
42.01
2,626.9528
1,890.46
38.04
53.32
32.86
1,130.87
300.18
-
operators in each project indicated that at least some centers
had problems with power supply, connectivity, and slow response. A
significant proportion (20 to 50 percent) of operators indicated
that problems in these three areas are encountered sometimes (as
opposed to never, rarely, often, and always). Hardware and
software, in comparison, were seen to be more robust.
In terms of agency impact for Chilean cases, the SII (Tax Online
system, Chile) indicated a significant increase in the efficiency
of internal work processes. Implementation of ChileCompra has
resulted in increased competition from suppliers. The total number
of opportunities for businesses published on the portal expanded by
75 percent in 2lD05 from the previous year. For example, micro,
small, and medium enterprises transacted 47 percent of all
transactions in 2005, which was 8 percent higher than in 2004.
Government departments found the system was easy to use and
reported significant gains in efficiency in the purchase of p