Utilizing Web 2.0 in SOA for Utilizing Web 2.0 in SOA for Enhanced Data Sharing Among Enhanced Data Sharing Among Federal Agencies Federal Agencies Nick Duan, Ph.D. Sr. Software Architect McDonald Bradley, Inc. Oct. 2, 2007
Dec 31, 2015
Utilizing Web 2.0 in SOA for Enhanced Utilizing Web 2.0 in SOA for Enhanced Data Sharing Among Federal AgenciesData Sharing Among Federal Agencies
Nick Duan, Ph.D.Sr. Software ArchitectMcDonald Bradley, Inc.
Oct. 2, 2007
MBI At a GlanceMBI At a Glance
• Information technology firm, founded in 1985
• Headquartered in Herndon, VA• Focus on DoD, IC, and Federal Civil• About 300 associates
– Experts and Industry leaders in enterprise technology and subject matter expertise
• Top Secret Facility clearance with many cleared professionals
CONTRACTOR OF THE YEAR
MBI Core Competency AreasMBI Core Competency Areas
Service-Oriented Enterprise
Data Interoperability
Information Assurance
Enterprise Governance and Management
Overview
• The Ultimate Quest of Sharing Data and Services• SOA Implementation – The Scalability Problem• SOA Enablement via Web 2.0
– Simple, Agile, Efficient
• Current Web 2.0 Technologies• A Web 2.0-based Framework for Enhanced Data
Sharing• A Sample Case Study• Future Tasks• Summary and Conclusions
The Ultimate Quest: Sharing Data and Services
• Millions of data records are published on the Internet by the government– Census data, weather data, housing data, map
data, budget data, airport data, etc..• Data utilization is limited
– Manual search and retrieval, lack of standards and quality control, limited corporation among agencies, long development cycles
• Unleash the potential– A robust and scalable SOA framework to add
values beyond search
General Concepts in SOA for Data and Service Interoperability
• Data Structuralization and Standardization– Data format, metadata standards, authoring tools
• Service Standardization– Protocol standard at transport level and message level
• Infrastructure Services– Search and Discovery– Service Registration– Metadata Registry and Discovery
• Security– Transport level and message/content level
• Governance Rules and Policies– Getting the right data at the right time
The SOA Scalability Problem
• Most SOA implementations today are based on Web Services with complex WS-* stack– Out-of-the-box adoption of the commercial SOA
models and products– Complexity: Distributed Object Computing (DOC)
model does not scale across enterprises– Centralized service registry and repository with
complex governance policies suitable for developing monolethic, closely-coupled systems
– Topdown, ”Creational” development – Lack of overall and consistent conceptual data models
across multiple domains• $Millions spent with limited results
Using Web 2.0 to Achieve Simplicity, Agility and Efficiency
• What is Web 2.0?– Collection of web technologies that leverage existing
web tools and infrastructure• Asynchronous JavaScript and XML (AJAX)• REpresentational State Transfer (REST)• Really Simple Syndication (RSS)• Rich Client Applications, Semantic Web, Wiki, Blogs, etc
– Client-side and Server-side Data Mash-ups• Examples: GoogleMaps, Yahoo Maps, Yahoo!Pipes,
Amazon, eBay, MyFaceBook• Key Advantages of Web 2.0
– Data and service mash-ups at various levels, enabling non-transactional data retrieval
– De-centralized, bottom-up, evolutional approach to enable rapid service offering
Asynchronous JavaScript and XML
• Microsoft introduced XMLHttpRequest in IE in late 90’s– Allows JavaScript object to connect with back-end web
servers directly– Enables partial update of web pages from multiple web sites – Simple, efficient object serialization via JSON
<script type=”text/javascript”>
........
</script>
XMLHttpRequestXMLHttpRequest
Web ServerWeb Browser
Server-sideComponent
Server-sideComponent
Server-sideComponent
Full Update
Partial Update
REpresentational State Transfer
• As an architecture style first introduced by Roy Fielding in 2000
• Basic design concept of a good protocol: – Separating transport control from content!– Utilizing the full spectrum of HTTP methods:
• POST, GET, PUT, DELETE– Representing states via resource URLs
• No need for cookies
• Resource-Oriented Computing (ROC): a virtual system comprised of multiple, interconnected web resources
REST Examples
• Retrieving part list info using GET (http://www.parts-depot.com/parts)
<?xml version="1.0"?><p:Parts xmlns:p="http://www.parts-depot.com" xmlns:xlink="http://www.w3.org/1999/xlink"> <Part id="00345" xlink:href="http://www.parts-depot.com/parts/00345"/> <Part id="00346" xlink:href="http://www.parts-depot.com/parts/00346"/> <Part id="00347" xlink:href="http://www.parts-depot.com/parts/00347"/> <Part id="00348" xlink:href="http://www.parts-depot.com/parts/00348"/></p:Parts>
• Retrieving individual part (http://www.parts-depot.com/parts/000345) <?xml version="1.0"?>
<p:Part xmlns:p="http://www.parts-depot.com" xmlns:xlink="http://www.w3.org/1999/xlink"> <Part-ID>00345</Part-ID> <Name>Widget-A</Name> <Description>This part is used within the frap assembly</Description> <Specification xlink:href="http://www.parts-depot.com/parts/00345/specification"/> <UnitCost currency="USD">0.10</UnitCost> <Quantity>10</Quantity></p:Part>
Benefits of Using REST
• Statelessness: Clients are completely de-coupled from the Server
• Uniform Interface: HTTP GET, POST, PUT and DELETE• Uniform State Representation: Define a virtual machine
whose states are represented by interconnected resources via URL
• Unrestricted Scalability: The Web is the limit
Data and Service MashupsData and Service Mashups
• Wikipedia: A mashup is a web application that combines data from more than one source into a single integrated tool
• In general, mashup is the concept of integrating data from multiple data sources to create presentation in a new dimension– Data mashup for presentation– Service mashup for reuse
• Client-side mashup– JavaScript, HTML/DHTML, AJAX, AdobeFlash, Customized APIs
• Server-side mashup– Proxy or gateway services providing data mapping, conversion,
and mediation functions
It's all about Mashups
Server-site MashupsServer-site Mashups
Client-side MashupsClient-side Mashups
Federal Data ProvidersFederal Data Providers
Client vs. Server-side Mash-ups
• Client-side mash-ups: data mash-ups within a browser– Simple, easy, cost-effective, end user driven– Limited functionality, primarily presentation use only,
constraint by security
• Server-side mash-ups: value-added, aggregated services for data integration– Feature rich, sophisticated functionality, less security
constraints– Complex, expensive, business driven
• Create the best combination to achieve maximum effectiveness and reuse
• Server-centric• Thin/simple client• SOAP, WSDL, UDDI• Heavy WS-* stack • Standard-committee
driven• Top-down, creational• Long dev cycle
Moving SOA to a New Dimension
• User/business-centric• Rich client• REST, AJAX, RSS,
REST Registry • Leveraging existing
HTTP/HTTPS • User and community
driven• Bottom-up, evolutional• Short development
cycle
From Web 1.0/SOA 1.0 From Web 1.0/SOA 1.0 to Web 2.0/SOA 2.0to Web 2.0/SOA 2.0
SOA Landscape with Web 2.0
FAA NOAA Census Bureau
USGS States Local
End Users
Aggregated Content and Service Providers
Open Infrastructure Services
YahooPipes
GoogleMaps
Amazon S3
Programmable Web
Yahoo News
Google RSSInternet
Data Providers
Open Source
A Case Study
• Case Scenario – Emergency Planning – In case of a category 5 hurricane, which area
(states/counties) may be affected– Which hospitals are to be evacuated?– Which airports may be used for delivering emergency
supplies?
• Data Contents and Providers– Real-time forecast data in RSS from NOAA (NWS, NHC)– County data from US. Census Bureau in CVS– Airport information from FAA in Excel/CVS– Hospital information from AHD in Excel
Plant 1 Plant 2 Plant 3 Plant 4
0
2
4
6
8
10
12
Estimated Generators Needed
Category 3
Category 4
Category 5
Interaction of Services
Aggregation & Mediation Service
Data MashupsService Registry and Discovery
21
Rich User Functions beyond Search
Geo-spatial Analysis: Locate the hospitals in the impact region
Plant 1 Plant 2 Plant 3 Plant 4
0
2
4
6
8
10
12
Estimated Reduction in Electricity
Category 3
Category 4
Category 5
Converting GIS data intoGoogle-compliant lat/long
Converting/Correlating hospital data AJAX-specific format
Statistical Analysis: Estimated number of evacuees per hospital
Analytics capabilities provided by service aggregation
Aggregated Service
Future TasksWhat are needed for SOA 2.0
Future TasksWhat are needed for SOA 2.0
• Aggregated services to enable rapid data and service integration– Data mapping, mediation,
• Government-sponsored infrastructure services to meet specific need of agencies– Service Registry and Discovery– Standard Metadata Registry– Standard cross domain taxonomies– Security if needed
• Government standard for data quality– Tolerable error rate, availability, security, etc.
Summary and Conclusions
• The majority of data published by the federal government have not been fully utilized
• Vast potential of data and service sharing can be unleashed via SOA
• A simple, agile and efficient SOA framework can be achieved using Web 2.0 technologies for enhancing data and service interoperability
• Data and service mashups are the key towards rapid development and service offerings