A web-based Census of services: an ISTAT evolutionary study Annalisa Cesaro, Bernardo Palazzi, Manlio Paterniti, Paola Ranaldi 5/03/2013
A web-based Census of services:
an ISTAT evolutionary study
Annalisa Cesaro, Bernardo Palazzi,
Manlio Paterniti, Paola Ranaldi
5/03/2013
1. The Italian experience in numbers
2. The Italian experience: the web architecture
3. The census web technologies
4. The Census Management System (CMS): a metadata-driven system
5. CMS: a functional overview
6. CMS: building blocks for a cost effective improvement
7. What is SOA?
8. SOA as a Web Service Architecture
9. SOA as a Web Service Architecture in Census LAN
10. SOA criticalities
11. SOA-based web solution ISTAT exploration: technical feasibility
12. SOA-based web solution ISTAT exploration: service overhead
13. SOA-based web solution ISTAT exploration: network latency
14. Conclusions
CENSUS TODAY
SOA PROPOSAL
ISTAT STUDY
2010-2012 online Census Surveys:
• 60 thousand forms for 2010 Agricolture Census (3%)
around 7000 census operators
• 8,5 million forms for 2011 Population and Household Census (one third
directly filled by citizens) (33%)
around 100000 census operators
• 420 thousand forms for 2012 Industry and Services Census (54%)
around 3000 census operators
The Italian experience in numbers
The Italian experience: the web architecture
Census network
internet
internet
Census Operations
Management System (CMS)
Online questionnaire (OQ)
Documentation System (DOC)
Oracle
MySQL
The census web technologies
MySQL
GUI QO
Actions
Services
DAOs
Entities
Not-SOA
GUI CMS
Actions
Services
DAOs
Entities
Web site
Typo3
PHP
Oracle
Process for Postal System on its
own IT
Microdata Processing for
dissemination on its own
IT
Microdata Processing for
first evaluations on its
own IT Replicated
mappings
Replicated
mappings
connection
Replicated
mappings
connection
or BATCH
sharing of
data
Replicated
mappings
connection
Replicated
mappings
connection
or BATCH
sharing of
data
The Census Management System: a metadata-driven
system
• Text are stored as metadata;
• The questionnaire processing rules are stored in metadata;
The questionnaire processing has been modeled as a finite state
machine. A triggering event determines a questionnaire state transition.
The state transition tables are a key element in CMS database.
1 | delivery 31 | arrival
registered
arrival registration
function (function 1)
Function 1 Function 2
State 1 State 31 State 1
State 2 State 2 State 2
Output
State 1 Delivery
… …
State 31 Arrival
registered
The Census Management System: a functional overview
A Census Management system:
• manages census operators: creation of the operators networks (roles
and responsabilities);
• eventually integrates different data sources for minimizing errors;
• trace the questionnaires life cycle, by handling the state flow of each
questionnaire;
• monitoring of the survey process by supervising of all the operations
performed on each questionnaire; SEPARATE TABLES –
UPDATED PERIODICALLY –
HEAVY PROCESS
BATCH PROCESSING FOR
UPLOADING AND
VALIDATING IMPORTED
DATA FROM DIFFERENT
SOURCES – HEAVY
PROCESS
The Census Management System:
building blocks for a cost effective improvement
A WEB BASED CENSUS OF SERVICES WHICH IS
BETTER CONNECTED TO BOTH INSIDE AND OUTSIDE
BETTER HEAVY PROCESSES ON
DEDICATED DB SERVERS FOR
PERFORMANCE
BETTER NONE CODE REPLICATION
FOR MAINTENANCE BETTER EASILY INTEROPERABLE
SYSTEMS
STRONG METADATA - DRIVEN
APPROACH
WELL DEFINED FUNCTIONAL
PROCESSES
What is SOA?
SOA
Front
end Services
Service
Repository Service
Bus
Implementation Interface
Busines
s
Logic Data
GUI
Actions
Services
DAOs
Entities
Not-SOA
Web
Application
…
NO CODE
REPLICATION
INTEROPERABILITY
TECH NEUTRALITY
SOA as a Web Service Architecture
“A Web Service is a software system designed to support interoperable
machine-to-machine intercation over a network. It has an interface
described in a machine processable format (specifically WSDL). Other
systems interact with the Web Service in a manner prescibed by its
description using SOAP messages, typically transported on HTTP with an
XML serialization.” w3c specification
Security Reliability Transaction
messaging
XML
M
E
T
A
D
A
T
A
SOA as a Web Service Architecture in Census LAN
DB
Web Service CLIENT
REQUEST/RESPONCE
SOAP/XML
Container
Web Service SERVER
DATA & METADATA
DATA & METADATA
SOA
Front end –
Web service
CLIENT
Web
Service
SERVER
Service
Repository Service Bus
SOA criticalities Distributed objects have a number of architectural
challenges and others point out:
• Problems introduced by latency and unreliability of
underlying transport;
• The lack of shared memory between the caller and
object;
• The problems introduced by partial failures
scenarios;
• The challenges of cuncurrent access to remote
resources;
• Overhead in terms of serialization/deserialization
and service handle ;
• Great amount of data;
NET LATENCY
EXCEPTION
CLASSIFICATION
PERFORMANCE
TECHNICAL FEASIBILITY
IN CENSUS CONTEXT
Client
program
WS handle
XML SOAP
message
serialize deserialize
XML SOAP
message
network
XML SOAP
message
XML SOAP
message
deserialize serialize
SOA-based web solution ISTAT exploration:
technical feasibility
Oracle
GUI
Actions
SOA
Web
Service
Services
DAOs
Entities
Menu item
Function 1
Function 2
Function 3
Function 4 1:1 RELATIONSHIP WITH
WEB SERVICE OPERATION TECHNICAL FEASIBILITY
IN CENSUS CONTEXT
WAR file
WAR file
SERIALIZABILITY
THROUGH EXTERNAL
JAX-WS BINDING
NO CIRCLE REFERENCES
IN ENTITIES
SOA-based web solution ISTAT exploration:
service overhead Machine A
Front end WAR file
Web Service WAR file
Oracle
Machine B
NO NETWORK LATENCY
TESTS UP TO 10000
RECORD EXCHANGED
SOA-based web solution ISTAT exploration:
network latency Machine A
Front end WAR file
Oracle
Machine C
YES NETWORK
LATENCY
Machine B
Web Service WAR file
TESTS UP TO 10000
RECORD EXCHANGED
Conclusions
- First classification of advantages of a
SOA-based web architecture in survey context,
thanks to the metadata-driven features of actual
web-based systems and to the modular
functional nature of them;
- Piloting applications for:
- technical feasibility testing;
- first classification of SOA
criticalities in census context;
- Performance evaluation for
design/monitoring/routing
purpouses;
GOOD RESULTS
PRODUCES FURTHER
EXPLORATION NEED!
Thank you