eXtc: XML Technology for Caché Overview Document · SOAP/WSDL-based Web Services, without the need for mapping to Caché Objects or upgrading to the most recent versions of Caché.

eXtc: XML Technology for Caché

Overview Document

Version 4.0.1a

======================================================

IMPORTANT NOTE

Current Status: 12 October 2005

extc is now distributed as a free product. See our web site (http://www.mgateway.com) for details.

Version 4.0’s main improvements include:

• Support for SOAP Headers within the automatic WSDL/SOAP Client functionality;

• Additional macro DOM API functions that perform a range of useful DOM transformations;

• Various internal fixes to improve stability and performance;

eXtc has also been enhanced to cater for complex and difficult situations that may be encountered whenexchanging XML via HTTP and HTTPS with remote servers. These facilities allow eXtc to provide XMLsolutions where other technologies may not be able to cope.

eXtc : XML Technology for Caché: Overview Document. Version 4.0.1a: 12 October 2005. 2001-2005, M/Gateway Developments Ltd. All Rights Reserved

1

eXtc : XML Technology for Caché

Overview Document

Version 4.0.1a

What is eXtc?

eXtc is a complete implementation of the W3C XML DOM standard for the Cachédatabase. It consists of a set of components, all of which are written in CachéObjectScript, and which will run on Caché version 5.x. eXtc is fully conformant with allthe relevant W3C standards and recommendations for XML. It requires no add-oncomponents except, optionally

• Caché Server Pages (CSP), WebLink or the m_php PHP gateway, to allow a Cachésystem to act as an HTTP-based web service or, more generally, an XML server

• A third-party add-on SSL proxy such as Delegate or OpenSSL to allow a Cachésystem to act as an XML client to a secure HTTPS-based web service or XML server.

In developing eXtc, the intention has been four-fold:

• to open up access to legacy Caché applications and databases via the now universallyaccepted XML-based standards such as SOAP and WSDL. Caché dominates thehealthcare application marketplace, and significant sections of the financial servicesindustry use Caché. eXtc provides a simple, rapid and very high performancesolution for making existing Caché applications accessible via, for example,SOAP/WSDL-based Web Services, without the need for mapping to Caché Objects orupgrading to the most recent versions of Caché.

• to leverage and exploit the performance, scalability and rapid applicationdevelopment capabilities of the Caché environment, and make Caché the database ofchoice for the rapidly emerging web service marketplace.

• to provide a comprehensive and high performance native XML-DOM basedenvironment within Caché, with which XML document manipulations that are moresuited to the random access nature of the DOM may be carried out. Examples includeXSL-FO, SVG and HTML document management tasks, the latter requiring theHTML to XHTML converter that is included in the eXtc Web Developer product.

• to position Caché as a native XML database, for example to allow Caché to be usedfor document management.


2

The eXtc Core Architecture

eXtc has, at its core, two key components:

• a fully W3C conformant implementation of the Document Object Model (DOM)Level 2 Core Specification Version 1.0 [W3C Recommendation 13 November 2000]

• a fully W3C conformant XML validating Processor (Parser) based on the followingstandards:

• Extensible Markup Language (XML) 1.0 (Second Edition) [W3CRecommendation 6 October 2000: REC-xml-20001006] – This includesDocument Type Definition (DTD)

• Namespaces in XML [W3C Recommendation 14-January-1999: REC-xml-names-19990114]

• DOM Level 3 for Abstract Schemas and Load and Save Specification [W3CWorking Draft 7 June 2001: WD-DOM-Level-3-ASLS-20010607]

• XML Schemas:

a) XML Schema Part 0: Primer [W3C Recommendation, 2 May 2001: REC-xmlschema-0-20010502]

b) XML Schema Part 1: Structures [W3C Recommendation, 2 May 2001: REC-xmlschema-0-20010502]

c) XML Schema Part 2: Datatypes [W3C Recommendation, 2 May 2001: REC-xmlschema-0-20010502]

All of eXtc’s components have been written using the Caché ObjectScript programminglanguage, and the DOM is implemented directly in Caché using globals. eXtc includes anobject wrapper that allows all the API’s to be invoked via Caché’s object syntax. Extc isdesigned for use with Caché version 5.x, on all its supported hardware platforms. Thefigure shows, diagrammatically, the core of eXtc’s architecture, and these components aredescribed in more detail in the following sections.


3

The XML DOM

The DOM is the heart of eXtc and it is the focus on the DOM that gives eXtc itsextraordinary power and flexibility. The DOM has been specified by the W3C as an in-memory representation of any tagged document, and, as such, can be used to represent allfeatures and aspects of any XML document. Within this overview document, we willrefer to a DOM that is populated with an XML document as an “XML DOM”.

The DOM also specifies a standard set of methods and attributes that are used to build,navigate and modify an XML document programmatically. The DOM can beprogrammatically converted back into the corresponding XML document via a well-defined eXtc DOM method invocation.

You can therefore consider an XML document, ie the data represented using XML tags,to simply be an interchange format – only when an XML document is being imported intoeXtc or exported from eXtc will it exist in this format. Within eXtc, the DOM is theformat that is used for representing and manipulating an XML document.

Whereas most other industry implementations of the DOM are implemented in memory,eXtc can take advantage of Caché’s native data structure – globals. In fact the DOMmaps extremely naturally and efficiently onto globals, and the highly optimised in-memory caching of globals within the Caché environment provides DOM performancethat is approaching (and in some situations faster than) an in-memory DOMimplementation. Indeed one of the usual criticisms of the DOM is that its implementationis memory intensive, a problem that becomes worse as the size of the XML documentincreases. By mapping the DOM to globals, eXtc avoids all such deficiencies and


4

eXtc Parser & DOM

<INCOME><SALARY><YEAR>1999</YEAR><AMOUNT>20000.00</AMOUNT></SALARY><SALARY><YEAR>2000</YEAR><AMOUNT>22000.00</AMOUNT></SALARY><TAX><AMOUNT>20.00</AMOUNT></TAX></INCOME>

Incoming/OutgoingXML Document/Dataset

eXtc Parserconforms to

W3C XML 1.0& XML Namespaces 1999

Structural& syntactical

errors

eXtc DOMBuilder

XML DOMConforms to W3C DOM Level 2 CoreSpecification v1.0

eXtc DOMAPI functions

Interchangeformat Internal storage

and manipulation format

Convert DOMto XML

Programmer access usingCaché ObjectScript or

Caché objects

bottlenecks, and it is quite feasible to build and maintain huge XML documents, and,furthermore, store them persistently in DOM format.

Very few database environments are capable of maintaining and storing an XMLdocument in its native DOM structure, and none so efficiently and naturally as Caché.The DOM does not easily map onto, for example, relational database two-dimensionaltables, and so a relational database environment cannot make use of the extremelyeffective and powerful API provided by the DOM.

eXtc on the other hand, allows the Caché database environment to be treated as a nativeXML database, in which XML documents can be stored permanently in their W3C DOMstructure. You can also make use of all the powerful database management andtransaction processing features of Caché – makingeXtc in combination with Caché a verypowerful native XML database environment. With eXtc, any number of XML documentscan be stored in a Caché database, limited only by your disk storage, and all can bemanipulated and maintained via the standard XML DOM API. eXtc is one of only veryfew implementations of W3C’s DOM Level 2 specification, and as such it provides fullsupport for namespaces (essential, for example, for XML schema validation and for fullvalidation of WSDL documents and SOAP requests and responses). Indeed we believethat eXtc was the first DOM implementation to implement any of the ASLS DOM Level3 draft recommendations that provide extensions to the DOM to allow storage andmanipulation of DTDs and XML Schemas.

The XML DOM API methods, as specified by the W3C, are implemented as a set ofCaché methods, and these can be used to map data in and out of a DOM, to and fromexisting Caché database structures (eg in legacy applications). The DOM APIs are alsowrappered as Caché Objects, and all the DOM properties and methods are exposed andavailable as specified by W3C.


5

eXtc DOM ManipulationXML DOM(s)

eXtc DOM API functions

Create newDOM instance

Functions forModifying DOM

Functions fornavigating and

accessing DOM contents

Conforms toW3C DOM

Level 2 CoreSpecification

v1.0{

Programmer access interfaces usingCaché ObjectScript or Caché Objects

It is therefore a simple task for a Caché programmer to build an XML documentprogrammatically via the DOM API methods, or to navigate and access the informationreceived in an incoming XML document. eXtc automates all the necessary functionality(document serialisation and validation) needed to import and export XML documents,leaving the programmer the simple task of building and/or accessing instances of XMLDOMs.

All of eXtc’s value-added functions (eg the XML Client, XML Server, WSDL/SOAPWeb Services Client and Server and the XPath support are all ultimately implementedusing the core DOM API methods. eXtc therefore represents a truly layered product, andit is this that gives it its robustness and flexibility.

The eXtc XML Processor (Parser)

A critical step in importing an XML document is its conversion to the DOM format.Before this can be done, a number of steps, some optional, must be carried out:

• parsing of the incoming XML document, breaking it down into its constituentelements, attributes, data etc.

• performing validation to ensure that the XML document is well formed. Anystructural deficiencies (as defined in the W3C Recommendation for XML 1.0 (2nd

Edition), XML Namespaces and XML Schemas) cause rejection of the document andno further processing will take place. Comprehensive error reporting is provided,referenced against the W3C Recommendation document.

• Optionally performing validation of the XML document against a Document TypeDefinition (DTD), either internal (ie embedded within the XML document) orexternal (ie as a separate file or document).

• Optionally performing validation of the XML document against an XML Schema.

These functions are all provided by the eXtc XML Processor (Parser). The Processor(Parser) is written in Caché ObjectScript, and requires no additional add-on third partycomponents.

The performance of the eXtc XML processor (parser) has been of key consideration in itsdesign, but this has not been at the expense of its conformance with the W3C standards.Indeed the eXtc Processor (Parser) has been found to identify some error conditions thatare missed by the XML processor (parser) within Microsoft’s Internet Explorer v5.5. Weare unaware of any other XML processor (parser) that runs natively within the Cachéenvironment, and that meets a similar combination of performance and conformance withthe XML standards.


6

The XML DOM Builder

eXtc includes, at its core, one further component – the high-performance DOM Builder.This sits between the XML Processor (Parser) and the DOM, and is called by theProcessor (Parser) to build a live instance of the DOM for the document that is beingparsed.

Timings for a 1Mb XML document have shown that, on some platforms, eXtc wascapable of parsing it and building an instance of a DOM significantly faster thanMicrsoft’s Internet Explorer v5.5.

Importing and Exporting XML to/from a Caché System

XML is primarily thought of as a data interchange format, allowing platform independentexchange of data. A critical part of any XML environment is therefore the import andexport mechanisms whereby a server can send or receive data exchanged in XML format.

XML does not, itself, define or dictate anything about its transport, and indeed anytransport mechanism that is supported by two servers can be used to exchange databetween them. However, there are a number of methods and transport protocols that arebeing adopted throughout the industry for XML exchange, in particular the use of Internetprotocols such as FTP, SMTP and HTTP. Of these, the latter, HTTP (and its securevariant, HTTPS), is emerging as the most important transport protocol for XMLdocument exchange, and indeed the combination of XML and HTTP is the underpinningof the rapidly emerging Web Services architectures.

The diagram below summarises the main ways in which XML can be imported andexported via eXtc.


7

Importing an XML document

It should be clear by now that the endpoint of importing an XML document into the eXtcenvironment is its instantiation as a DOM. A DOM can be created in one of two ways:

• by the eXtc Processor (Parser) and DOM builder, from an XML document

The XML document being parsed and converted to a DOM can have come from anumber of sources:

• a text file imported to the Caché server by some means, or as a result of an XMLeditor package;

• a Caché global containing lines of raw XML;• via HTTP (or HTTPS), received by the Caché system via WebLink, CSP or PHP

(the latter using the M/Gateway Developments m_php gateway)• via some other transport protocol (eg FTP, POP3)

• programmatically via the DOM API methods from within the Caché programmingenvironment


8

eXtc XML Import/Export

<INCOME> <SALARY> <YEAR>1999</YEAR> <AMOUNT>20000.00</AMOUNT> </SALARY> <SALARY> <YEAR>2000</YEAR> <AMOUNT>22000.00</AMOUNT> </SALARY> <TAX> <AMOUNT>20.00</AMOUNT> </TAX></INCOME>

XML Document/Dataset

Parser DOMBuilder

Convert DOMto XML

Send usingHTTP POST

To remoteHTTP server

WebServerCSP/

WebLink

HTTP POSTfrom remote

server

Text File

Text File

XML DOM

Otherexport method

Otherimport method

CachéSystem

DOM APIMethods

ProgrammerAccess

Exporting an XML document

To export an XML document, the eXtc function is invoked that converts a DOM into itsequivalent XML tagged document format (outputDOM()). The XML document streamthis creates can be routed:

• to create a text file (which can be exported in some way);• to create a Caché global;• via HTTP (or HTTPS), either:

• as an HTTP request to a remote XML server, sent directly from Caché;• as an HTTP response, returned to a remote XML client via WebLink or CSP;

• via some other protocol such as FTP or SMTP.

Validation using DTDs and Schemas

In an earlier section it was noted that the eXtc Processor (Parser) offers optionalvalidation of an incoming XML document against its DTD or XML Schema. DTDs andSchemas are alternative ways of defining formally the structure and content of an XMLdocument. Schemas are expected to replace the use of DTDs, primarily because:

• they are also XML-based (and can therefore be parsed and manipulated just as anotherXML document);

• they are more far reaching in their capabilities of specifying an XML document’sstructure and content, for example by allowing data-typing and namespaces to bespecified.

eXtc supports both DTDs and Schemas (though full validation against schemas is notcurrently supported). Essentially there are three alternative scenarios:

• XML document with external DTD

In the majority of situations where DTDs are used, the DTD will exist as a separatefile, created as a “one-off” specification exercise. An incoming XML document canthen be validated against the “external” DTD. Validation against an external DTD isshown diagrammatically in the figure below:


9

The structure and syntax of DTDs differs from XML and therefore there aredifferences in their parsing and internal storage. To harmonize this, eXtc uses W3C’sdraft specification for the Level 3 Abstract Schema (ASLS) DOM which provides astandard for storage and manipulation of DTDs and XML Schemas. We believe this isone of the first working implementations of this emerging standard. In this document,we will refer to an instance of a DTD or an XML Schema in the Abstract SchemaDOM as an AS DOM.

The steps involved are as follows:

• Import or create the DTD file. This will be a simple text file containing the DTDfor a specific XML document. You may decide to use a DTD editing tool such asXMLSpy, or alternatively a business partner may have sent you a DTD file (eg byemail) that defines XML documents they will be sending you or expecting toreceive from you.

• Parse the DTD file and build an instance of an AS DOM. eXtc supports theASLS DOM API methods that allow you to manipulate, edit or modify an ASDOM, and these could, in fact, be used to build a DTD from scratch.

• When an XML document for which you have a DTD is imported and parsed, youcan switch on the optional DTD validation. The XML document is then checkedby the processor (parser) for conformance against the specification held in the ASDOM. Any non-conformance is reported as a set of errors. Only if the incomingXML document conforms to the DTD will it be converted to an instance of anXML DOM.


10

External DTD Processing and Validation

Parser DOMBuilder

XML DOM

<!DOCTYPE INCOME SYSTEM “Income.dtd”><INCOME> <SALARY> <YEAR>1999</YEAR> <AMOUNT>20000.00</AMOUNT> </SALARY> <SALARY> <YEAR>2000</YEAR> <AMOUNT>22000.00</AMOUNT> </SALARY> <TAX> <AMOUNT>20.00</AMOUNT> </TAX></INCOME>

Imported XML Document/Dataset

DTD Parser DTD DOMBuilder

AS DOM

Imported DTD

Interchange formatInternal storage/

manipulation format

<!ELEMENT INCOME (SALARY+,TAXES)><!ELEMENT SALARY (AMOUNT,YEAR)><!ELEMENT AMOUNT (GROSSAMOUNT,NETAMOUNT)><!ELEMENT GROSSAMOUNT (#PCDATA)><!ELEMENT NETAMOUNT (#PCDATA)><!ELEMENT YEAR (#PCDATA)><!ELEMENT TAXES (TAX)><!ELEMENT TAX (AMOUNT,YEAR)><!ELEMENT AMOUNT (GROSSAMOUNT,NETAMOUNT)><!ELEMENT GROSSAMOUNT (#PCDATA)><!ELEMENT NETAMOUNT (#PCDATA)><!ELEMENT YEAR (#PCDATA)><!ATTLIST TAX taxyear CDATA #IMPLIED>

Validation andconformance towell-formedness

constraints

Errors

Optionalvalidation

against DTD

ASLSDOM APIMethods

Programmaticmaintenance

of DTD}

• XML document with internal DTD

An XML document can, optionally, include its own DTD – in this case the XMLdocument is completely self-standing and self-specifying. Validation against such an“internal” DTD is shown diagrammatically in the figure below:

Once again, the DTD is parsed and built as an instance of an AS DOM.

The steps involved are as follows:

• Import or create an XML document that includes its own DTD• When the XML document is parsed, eXtc's Processor (Parser) recognises the

internal DTD and separates it from the rest of the XML document.• The DTD component is parsed and eXtc builds an instance of an AS DOM.• If you have switched on the optional DTD validation, the rest of the XML

document is checked by the processor (parser) for conformance against thespecification that is now held in the AS DOM. Any non-conformance is reportedas a set of errors. Only if the incoming XML document conforms to the DTD willit be converted to an instance of an XML DOM.

An AS DOM that has been built from an internal DTD can, optionally, be saved inAS DOM format for future use. However, in most situations it will exist only duringthe parsing and validation of the incoming XML document, after which it will becleared from the Caché system – this is because subsequent instances of this XMLdocument are likely to always include an internal DTD.


11

Internal DTD Processing and Validation

ParserDOMBuilder

XML DOMImported XML Document/Dataset

with embedded DTD

DTD Parser DTD DOMBuilder

AS DOM

Interchange formatInternal storage/

manipulation format


constraints

Errors

Optionalvalidation

against DTD

<?xml version="1.0"?><!DOCTYPE note [ <!ELEMENT note (to,from,heading,body)> <!ELEMENT to (#PCDATA)> <!ELEMENT from (#PCDATA)> <!ELEMENT heading (#PCDATA)> <!ELEMENT body (#PCDATA)>]><note> <to>Tove</to> <from>Jani</from> <heading>Reminder</heading> <body>Don't forget me this weekend</body></note>

Internal storage/manipulation format

ASLSDOM APIMethods

Programmaticmaintenance

of DTD}

• XML document with a Schema

Schemas are XML documents that formally describe other XML documents.Schemas, like DTDs are parsed and stored as an AS DOM. An incoming XMLdocument can then be validated against the Schema specification held as an instanceof an XML DOM. The current version of eXtc does not fully support Schemavalidation, but the planned, future mechanism for validation against a Schema isshown diagrammatically in the figure below:

The steps involved will be as follows:

• Import or create the XML Schema document. This may have been created usingan XML Schema editing tool such as XMLSpy. Alternatively it may have beensent to you by a business partner, by any of the means described earlier forimporting XML documents

• Parse the Schema and build an instance of an AS DOM. The ASLS DOM APImethods that allow you to manipulate, edit or modify the AS DOM could thenoptionally be used. In fact, these could be used to build a Schema from scratch.

• When an XML document that contains a reference to its Schema is imported andparsed, you can switch on the optional Schema validation. The XML document isthen checked by the XML Processor (Parser) for conformance against the


12

Schema Processing and ValidationImported XML

Document/Dataset

Parser DOMBuilder

AS DOM

Imported Schema

Internal storage/manipulation format

<?xml version="1.0"?><shipOrder> <shipTo> <name>Tove Svendson</name> <street>Ragnhildvei 2</street> <address>4000 Stavanger</address> <country>Norway</country> </shipTo> <items> <item> <title>Empire Burlesque</title> <quantity>1</quantity> <price>10.90</price> </item> <item> <title>Hide your heart</title> <quantity>1</quantity> <price>9.90</price> </item> </items></shipOrder>

<xsd:schema xmlns:xsd="http://www.w3.org/1999/XMLSchema">

<xsd:element name="shipOrder" type="order"/>

<xsd:complexType name="order"> <xsd:element name="shipTo" type="shipAddress"/> <xsd:element name="items" type="cdItems"/></xsd:complexType>

<xsd:complexType name="shipAddress"> <xsd:element name="name" type="xsd:string"/> <xsd:element name="street" type="xsd:string"/> <xsd:element name="address" type="xsd:string"/> <xsd:element name="country" type="xsd:string"/></xsd:complexType>

<xsd:complexType name="cdItems"> <xsd:element name="item" type="cdItem"/></xsd:complexType>

<xsd:complexType name="cdItem"> <xsd:element name="title" type="xsd:string"/> <xsd:element name="quantity" type="xsd:positiveInteger"/> <xsd:element name="price" type="xsd:decimal"/></xsd:complexType>

</xsd:schema>

ASLSDOM APIMethods

Programmaticmaintenanceof Schema}

Parser DOMBuilder


constraints

Errors

Optionalvalidation

against Schema

as above

XML DOMErrors

specification held in the AS DOM. Any non-conformance is reported as a set oferrors. Only if the incoming XML document conforms to the Schema will it beconverted to an instance of an XML DOM (although some errors may be non-fataland for information only).

XML DOM Persistence

eXtc stores and maintains imported XML documents as instances of XML DOMs.DOMs are stored in a global and are persistent, unless explicitly deleted by the user.Some of eXtc’s packaged functionality (eg the XML/HTTP Client and Server) clearsdown transient DOMs automatically, but this is done using built-in explicit calls to theeXtc functions that delete the documents from the DOM. DOMs can be optionallylocked to prevent their accidental deletion.

Persistent DOM instances can be manipulated using the standard DOM API methods, butthe Caché Objects wrapper allows DOMs to be queried using Caché SQL. You can alsouse a partial implementation of XPath, a standard for manipulation and navigation ofXML documents.

It is eXtc’s unique combination of persistent DOM storage within the database and theXML DOM API implementation that transforms Caché into a very powerful native XMLdatabase technology. The Caché database environment itself is a fully-fledged databasemanagement environment that includes sophisticated networking, scalability, backup andtransaction processing capabilities. eXtc in combination with Caché therefore representsone of the most powerful native XML database environments available.

eXtc in Action

The eXtc DOM processor, parser and API methods provide the core XML infrastructure,and provides the foundation for a growing set of value added functions. Specifically:

• A complete library of functions for generating PDF documents programmatically,using the W3C’s XSL-FO standard and the Apache Project’s FOP processor.

• XML/HTTP Client and Server functionality• eXtc’s leading-edge WSDL/SOAP Client and Server functionality.

These are summarized below.

The eXtc FOP Library for PDF document creation

eXtc includes a complete library of functions, built on top of the eXtc DOM APImethods, for building and creating PDF documents programmatically. PDF is an idealstandard for generating printable documents in web applications, particularly where theeXtc : XML Technology for Caché: Overview Document. Version 4.0.1a: 12 October 2005. 2001-2005, M/Gateway Developments Ltd. All Rights Reserved

13

page layout must be precise, and not subject to rendering artifacts of the browser or theuser’s screen resolution.

The eXtc FOP Library is based on the W3C’s XSL-FO (Formatting Objects) standard,and relies on the Apache Project’s FOP processor, a free Java program that will convertan XML document formatted according to the XSL-FO standard into the correspondingPDF document. The document can be either saved to a file for later use, or pipeddynamically via a web page to the user.

One of the great benefits of the XSL-FO approach is that the FOP processor is availablefor Windows and Unix/Linux platforms, so the eXtc FOP Library provides a non-proprietary, platform-independent approach to PDF file creation that can be freelymigrated between systems without any code changes.

The eXtc FOP Library is documented in detail separately (see the documentation setincluded with the eXtc Installation Kit), and includes complete installation instructionsand worked examples.

The eXtc FOP library demonstrates the power and value of the DOM approach toprogrammatic generation of XML documents. Future versions of eXtc are expected toinclude support for further XML-based value-added technologies.

The eXtc SVG Library for Vector Graphics creation and maintenance

eXtc includes a complete library of functions, built on top of the eXtc DOM APImethods, for building and creating vector graphics documents based on the SVGstandard. The full set of SVG tags can be created through a set of methods that can beinvoked within your Caché applications. These can be used to create graphs and otherimages in, for example, your web applications.

XML exchange over HTTP, and eXtc’s Web Services Support

Probably the most important and significant potential use and application of eXtc relatesto the exchange of XML documents via the HTTP and HTTPS protocols. Thecombination of XML and HTTP (and by implication in the rest of this document,HTTPS) provides the basis for extremely rapid, effective, high performance and highlyscalable integration of systems and applications. XML in combination with HTTP is thebasis of the genre of distributed collaborative systems known as Web Services. eXtc incombination with WebLink, CSP or PHP, allows Caché systems to operate seamlesslywithin these environments. Indeed to other systems a Caché system will simply beperceived to be another Web Service, accessible by standard XML and HTTP interfaces,and whose operation behind the scenes is irrelevant externally.


14

The figure below shows diagrammatically the way in which systems can collaborate usingXML and HTTP.

HTTP is an ideal protocol in many ways for this type of interchange because:

• It is a ubiquitous, non-proprietary protocol, being the protocol that underpins theWorld Wide Web

• It is a fast and highly scalable protocol, as witnessed by the huge volumes andthroughput of large web sites on the Internet

• It is a simple protocol, easily generated and totally text based• It is based around the concept of discrete request/response pairs: a client sends a

request to a remote server, which returns a response to the client. This makes itsuitable for situations where one system (the “client”) needs to make a request forsome action to take place on another system (the “server”), the results of which arereturned by the “server” system to the “client” system.

Generically this type of server-server interaction is known as a Remote Procedure Call(RPC). RPC using XML and HTTP can be implemented in many ways:

• in its simplest form, two collaborating systems can mutually agree on the structureand content of matching pairs of XML documents: one for a request from the “client”system, one for the response from the “server” system. The agreement on therequired request and response XML documents will normally be defined in terms ofDTDs or XML Schemas.

The processing that must take place on the “server” system in order to create theappropriate response is the sole responsibility of the “server” system. Provided the“client” system sends a correctly structured request that matches a valid DTD or XML


15

XML Server-Server Data Exchange over HTTP or HTTPS

XML “Client” XML “Server”

Internet/Intranet

Web Server

1. XML request sent via HTTP

or HTTPS

2. XML requestforwarded to

server

2.1.

3.

3. XML responsecreated and sent

to web server

4.

4. XML responsereturned via HTTP

or HTTPS

Schema, and provided the contents of that request make sense and are valid, the“server” system can run some process and package up the results as the appropriateresponse XML document. Once the “server” system has dispatched its response it hasno further interest in what takes place on the “client” system. Neither the “client” nor“server” systems need to know anything about each other apart from the URL(s)needed to send the HTTP-based request and the specification of the request andresponse XML documents. The collaborating systems can be running on totallydifferent and otherwise incompatible hardware, different operating systems, differentprogramming languages and database management systems (DBMS). Provided theycan communicate via HTTP and provided they can both process XML documents,then they are ready to inter-operate using this simple RPC architecture.

• A much more scalable and extensible standard for XML-based RPC has now emergedand is being adopted very rapidly across the IT industry: the Simple Object AccessProtocol (SOAP). Although SOAP does not, in itself, mandate any particulartransport protocol, HTTP (and HTTPS) is being adopted as the primary protocol, forall the same reasons outlined above. SOAP’s real importance is this industry-wideadoption, and is the underpinnings, for example, for Microsoft’s .Net strategy, inaddition to the RPC-based Web Service strategies from IBM and Sun.

SOAP differs from the simple type of XML document exchange described above inthat it assumes that the “client” system is aware of “methods” that can be serviced bythe remote “server” system, and these methods have a published interface in terms ofinput parameters. The calling interface (ie the name of the method to be run remotelyand the appropriate input parameters to that method) are expressed in XML terms,according to a strict syntax defined by SOAP. The results generated by the remote“server” system are then packaged up as a SOAP XML response document, thesyntax and structure of which is strictly defined and agreed.

An additional standard, the Web Services Description Language (WSDL), itself anXML-based standard, provides a formal specification of a Web Service’s request andresponse interfaces. A WSDL document therefore contains all the informationneeded to automatically determine the calling interface for a SOAP request, and todetermine the resulting response contents. eXtc provides full support for WSDL,allowing:

• simple, automated creation of WSDL documents to describe your Web Services.These should be sufficient for an external client to discover and use your SOAPmethods;

• automated fetching and analysis of an external WSDL document, and automatedgeneration of the client interface code needed to access the SOAP methodsdescribed by the external WSDL document.

eXtc therefore provides full, W3C standards-compliant Web Services support,allowing a Caché server to be either a Web Services client or server, or both.


16

A key feature of SOAP is that it does not mandate how it is implemented behind thescenes: although the externally visible interface is in terms of methods and their inputparameters, the implementation on the remote “server” system does not have to be interms of objects and methods. On a Caché system the implementation for servicing amethod may actually be to run a legacy function that was written many years ago.Provided it can be invoked as a result of receiving the appropriate SOAP request, andprovided its results are correctly packaged up and dispatched as the appropriate SOAPresponse, then to the outside world the remote method has been invoked and theresults returned. The WSDL support provided by eXtc can make that legacy functionautomatically discoverable by external third-parties.

By implementing SOAP and WSDL interfaces, a Caché system or application cantherefore make itself available as a set of methods, callable potentially by any system(obviously subject to security constraints) via an industry-standard interface. Howthose methods are implemented behind the scenes is invisible to and irrelevant tosystems that invoke those methods remotely. The fact that the method(s) being calledare actually running on Caché platforms becomes irrelevant to the calling system;what is much more important and visible to the outside world is the fact that theseCaché systems return valuable data, and do so very fast. A significant benefit of eXtcis that it becomes easy and quick to implement SOAP interfaces for existing legacysystems. It can also become apparent to the outside world that the ability to react todemand for new SOAP-based services is very rapid whenever the back-end systemconcerned is running the Caché database.

As a consequence of breaking down an existing application into a set of SOAP-basedWeb Services, the whole role and nature of existing legacy Caché applications can becompletely reconsidered. It can become possible for others to provide their ownalternative front-ends to existing applications – indeed whole new application front-ends and user interfaces could then become grafted onto existing legacy applications(typically otherwise limited to access via dumb terminals). eXtc will make this bothpractical and straightforward, involving the minimum of re-engineering and withoutthe need to migrate to newer versions of Caché or to Caché Objects. Even insituations where migration to Caché Objects has taken place, eXtc can still provide avery rapid development environment for the implementation of Web Servicesinterfaces.

• Another XML-based RPC standard is XML-RPC. This is a very simple butextensible standard for defining RPC mechanisms, but lacks the self-discoverycapabilities afforded by WSDL. eXtc includes full support for XML-RPC, allowing aCaché system to act as an XML-RPC client and/or an XML-RPC server. WebLink,CSP or PHP can be used as the gateway service for eXtc-based XML-RPC serversystems.


17

eXtc XML Server

Underpinning eXtc's HTTP support for XML exchange is a ready-written set of functionsthat allow a Cache system to act as either an XML client or an XML server. Here wedescribe the XML Server function; in a later section we will describe the XML Clientfunction.

eXtc provides a fully packaged XML server function that automates the entireXML/HTTP RPC process, leaving the programmer to focus on the task of running theappropriate back-end process and creating a response DOM: the figure below summarisesthe function. The red boxes describe the functions automatically provided by the eXtcXML Server. The grey boxes indicate the only functions that a programmer must designand implement. Note that you can use either WebLink, CSP or PHP to trigger the eXtcXML Server.

The key to the eXtc XML Server is its configuration with Caché WebLink, Caché ServerPages (CSP) or PHP (using the m_php gateway from M/Gateway Developments).WebLink, CSP and the m_php gateway provide connectivity between a web server andCaché servers.

The full details of how these gatways work in conjunction with eXtc are describedelsewhere in the XML/HTTP documentation. However, in summary, whenever triggeredby WebLink, CSP or PHP, the eXtc XML Server does the following:


18

eXtc “XML Server” processing

Caché Server

Internet/Intranet

WebServer

User validationof contents of request DOM Data stored

in globalsOr Cacheobjects

eXtc Server: Triggered by WebLink

Receive incoming XML request;Parse XML and build DOM

Validate request against DTD/SchemaHTTP

orHTTPS

WebLinkor CSP

User processingto use request DOM

and create newresponse DOM

eXtc Server:Convert response DOM to XML,

wrapper as HTTP response and send

From/to remoteHTTP Server

• the incoming XML document is transferred into a global (^MGW), and the XMLServer parses and optionally validates it against either a matching DTD or XMLSchema;

• Provided no fatal parsing and validation errors have occurred, the Server then buildsan instance of an XML DOM containing the incoming XML document.

• It then dispatches out to a routine determined by the programmer, based on either theDTD name or XML Schema specified within the incoming XML document. In thecase of incoming SOAP requests, it automatically dispatches to separate SOAPprocessing functions (see later).

• At this point it is the programmer’s responsibility to determine what to do. Typicallythis will involve the following steps:• Extract the relevant data from the DOM for the incoming request, using the DOM

API methods. The programmer probably needs to perform some application-levelvalidation and security checks, based on the information that has been received.

• Run some processing routine(s) based on the incoming request data• Create and build new DOM for the outgoing response, once again using the DOM

API methods.• Quit, thereby returning control to the eXtc XML Server.• If any application-level errors occur during these programmer-controlled steps,

they are used to build an outgoing error DOM and the programmer should thenquit without further processing.

• The eXtc XML Server then streams out the contents of the response DOM as theequivalent XML document, properly wrappered as an XML response.

Note that the eXtc XML Server supports incoming requests and outgoing responses thatuse the HTTPS protocol. This is because, in fact, HTTPS is handled by the web serverautomatically and everything that happens behind WebLink is unaffected by the transportprotocol between the XML “client” and web server.

eXtc XML Client

eXtc provides a ready-packaged function for situations where a Caché server needs to actas an XML “client” making requests to an external XML server. The figure belowsummarises the way it works. The red box in the middle denotes the functions providedby the eXtc XML Client:


19

The eXtc XML Client function is used as follows:

• The programmer creates and builds a new instance of a DOM that contains theoutgoing XML request document details.

• The programmer then calls the eXtc XML Client function which automatically doesthe following:• It makes a TCP connection to the specified remote HTTP server, and streams out

the contents of the XML request DOM as the equivalent XML document, properlywrappered as an HTTP POST request.

• It then awaits the HTTP response from the remote server and extracts the XML (ifany), parses it and, optionally, validates it against the appropriate DTD or XMLSchema

• It builds a new instance of a DOM, containing the contents of the incoming XMLresponse.

• It then quits, returning control to the programmer• The programmer can then use the DOM API methods to access the data from the

incoming XML response document, and do whatever processing is appropriate.

Note that when acting as an XML Client, eXtc does not use WebLink, CSP or the m_phpgateway, because they have no role to play in this scenario. It is the Caché ObjectScriptprogramming environment that opens the TCP socket connection to the remote HTTPserver, sends the request across it and awaits and reads the returned response across itbefore closing the socket.

Caché has no built-in support for the Secure Sockets Layer (SSL), so HTTPS cannot bedirectly supported from within eXtc. In situations where HTTPS is required for the eXtcclient, it is recommended that a third-party SSL proxy such as DelegateeXtc : XML Technology for Caché: Overview Document. Version 4.0.1a: 12 October 2005. 2001-2005, M/Gateway Developments Ltd. All Rights Reserved

20

eXtc “XML client” processing

Internet/Intranet

HTTP

To/from remoteHTTP Server

HTTP

Build XMLDOM for

outgoing streamData storedin globalsor objects

eXtc Client: Connect to remote web server;Convert DOM to XML stream;Send XML as HTTP request;

Receive response;Parse incoming XML

Build new DOM

User processingof response DOM,

eg save data

Data storedin globalsOr objects

eXtc Client: Connect to remote web server;Convert DOM to XML stream;Send XML as HTTP request;


Validate against DTD/SchemaBuild new response DOM

(www.delegate.org) and/or OpenSSL (www.openssl.org) is used to convert to and fromHTTPS and redirect the requests and responses to the Caché server. The diagram belowsummarises how the eXtc XML Client can be used in an SSL environment.

The eXtc XML Client continues to communicate via HTTP, but is configured tocommunicate with the SSL Proxy. The SSL Proxy, in turn, is configured to redirectrequests to the remote HTTP server, and to convert the protocol to HTTPS. The returnedHTTPS response is converted back to HTTP and returned to the awaiting XML clientfunction.

eXtc SOAP/WSDL Web Services Functions

eXtc provides a ready-packaged set of functions for handling SOAP-based Web Services,whereby a Caché system can be either a SOAP client or SOAP server. The eXtc SOAPclient and server functionality is built on top of the standard eXtc XML Server and Clientdescribed above.

In order to make a Caché server act as a Web Services server, eXtc provides anautomated mechanism for building a WSDL document that describes the calling interfaceto the associated SOAP method, and also for describing the dispatch call to the Cachémethod, routine or function. The WSDL document can be fetched via a simple URL overthe web, and provides all the information needed for the remote client to invoke the WebService on the Caché server. The remote client may be a .Net client, or any standard WebService client.

eXtc's WSDL conformance has been tested using a number of third-party validation sites,and its interoperability with other standard WSDL clients and servers is exceptional.


21

eXtc “XML client” processingusing SSL

Caché Server

Internet/Intranet

HTTPSProxy

Build XMLDOM for

outgoing streamData storedin globalsor objects

eXtc Client:Connect to remote web server;Convert DOM to XML stream;Send XML as HTTP request;


Build new DOM

User processingof response DOM,

eg save data

Data storedin globalsor objects

HTTP

HTTPS

HTTPS

HTTP

eXtc Client:Connect to remote web server;Convert DOM to XML stream;Send XML as HTTP request;


Validate against DTD/SchemaBuild new response DOM

eg Delegate,OpenSSL

To/from remoteHTTP Server

In order for a Caché server to act as a Web Services client, eXtc provides the followingmechanisms:

• The ability (from within a Caché process) to fetch a WSDL document from a remoteserver (using an HTTP GET)

• Analysis of the WSDL document to determine the methods it describes and the inputsand outputs for those methods

• Automatic generation of a set of Caché ObjectScript methods that provide the callinginterface to the remote Web Service methods. These methods automatically generatethe appropriate SOAP request, send it to the SOAP Server, receive the SOAPresponse and convert it to a DOM ready for use by the user/programmer. The SOAPresponse may be analysed and manipulated using either DOM API methods or XPathqueries.

The mechanism used for fetching and saving a WSDL document is illustrateddiagrammatically below. Note that this mechanism ensures that repeated requests for aWeb Service make use of a locally cached and parsed version of a WSDL document.

Note that because the WSDL document is, itself, an XML document, eXtc stores it as aDOM, and it is in this DOM format that it is used for SOAP request/response automation.

It is envisaged that later versions of eXtc will support Universal Description, Discoveryand Integration (UDDI). This is an XML-based service that describes the services that acompany can provide, and provides pointers to the WSDL documents that then formallydescribe the SOAP interfaces for those services.


22

eXtc WSDL Import

<?xml version='1.0'?><definitions name="CATrafficService" targetNamespace="http://www.xmethods.net/sd/CATrafficService.wsdl" xmlns="http://schemas.xmlsoap.org/wsdl/" xmlns:soap="http://schemas.xmlsoap.org/wsdl/soap/" xmlns:tns="http://www.xmethods.net/sd/CATrafficService.wsdl" xmlns:xsd="http://www.w3.org/1999/XMLSchema"><message name="getTrafficRequest"><part name="hwynums" type="xsd:string"></part></message><message name="getTrafficResponse"><part name="return" type="xsd:string"></part></message><portType name="CATrafficPortType"><operation name="getTraffic"><input message="tns:getTrafficRequest" name="getTraffic"></input><output message="tns:getTrafficResponse" name="getTrafficResponse"></output></operation></portType><binding name="CATrafficBinding" type="tns:CATrafficPortType"><soap:binding style="rpc" transport="http://schemas.xmlsoap.org/soap/http"></soap:binding><operation name="getTraffic"><soap:operation soapAction=""></soap:operation><input><soap:body encodingStyle="http://schemas.xmlsoap.org/soap/encoding/" namespace="urn:xmethods-CATraffic" use="encoded"></soap:body></input><output><soap:body encodingStyle="http://schemas.xmlsoap.org/soap/encoding/" namespace="urn:xmethods-CATraffic" use="encoded"></soap:body></output></operation></binding><service name="CATrafficService"><documentation>Provides information on California Highway Conditions.</documentation><port binding="tns:CATrafficBinding" name="CATrafficPort"><soap:address location="http://services.xmethods.net:80/soap/servlet/rpcrouter"></soap:address></port></service></definitions>

HTTP GETRequest for

WSDL document

To remoteHTTP server

From remoteHTTP server

Parser

DOMBuilder

WSDLstored as

XML DOM

Glossary

API Application Programming Interface: the interface through which theDOM is navigated and manipulated programmatically

Caché A database managememt system and programming environment, sold byInterSystems Corporation

CachéObjectScript

InterSystems’ proprietary scripting language for the Caché database,compatible with the M scripting language but with extensions, forexample, to support Objects

Client System and/or software that initiates an interaction with a remote system

DOM Document Object Model: a programmable representation of a taggeddocument, such as an XML document

DTD Document Type Definition: a document that defines the specification foran XML document

FTP File Transfer Protocol: Internet protocol for exchanging files betweensystems

Global The persistent data structure used internally by Caché. Globals aresparse, hierarchical persistent arrays, whose dimensions are totallydynamic and do not need to be pre-declared.

HTTP Hypertext Transport Protocol: a text-based wire protocol, based on asimple request/response pair between a “client” and “server”

HTTPS Secure, encrypted version of the HTTP protocol

M The MUMPS programming language and database management system

Parsing Breaking down a document into its constituent elements, attributes,elements etc

POP3 Post Office Protocol: Internet protocol for mail-box servers, allowingstorage of emails on the server and interrogation and download of emailsby a client system

Proxy An intermediate system and/or software that sits between a client andserver and performs some form of translation of the request from theclient and/or from the server.


23

RPC Remote Procedure Call: the mechanism whereby a client invokes amethod on a remote server. The remote server returns the results back tothe client when the method has executed

Server A remote system that receives requests from clients, and which performsa transaction

SMTP Simple Mail Transfer Protocol: Internet protocol for sending messages,for example email, from a client system to a server

SOAP Simple Object Access Protocol: A non-proprietary XML-based standardfor RPC. Defined by a consortium of companies including Microsoft,and provides the RPC layer for various companies’ web servicestrategies (eg Microsoft .Net, Sun One, IBM)

SSL Secure Sockets Layer: a standard used within the World Wide Web toallow secure encrypted exchange of requests and responses betweenclient and server. SSL uses the HTTPS protocol for this purpose

W3C World Wide Web Consortium: a standards-making body for web-basedservices and technologies

Web Services Distributed collaborative systems, integrated using web protocols andtechnologies such as SOAP and HTTP

XML Extensible Markup Language

XMLDatabase

A database management system that stores and manages data as XMLdocuments

XML DOM A DOM that is populated with an XML Document

XML Schema An XML document that formally specifies another XML document interms of its structure and content

Validation The comparison of an XML document against it specification in either aDTD or XML Schema. Normally any discrepancies between the XMLdocument and the specification will result in errors being reported.Some of these errors may be fatal, in that they are so serious that theXML document has to be rejected as it is impossible to understandcorrectly and therefore to handle programmatically.

WSDL Web Services Description Language. An XML-based standard fordescribing Web Service methods and their calling and responseinterfaces.


24

eXtc: XML Technology for Caché Overview Document · SOAP/WSDL-based Web Services, without the need for mapping to Caché Objects or upgrading to the most recent versions of Caché.

Documents