1 Professional Services www.tibco.com Global Headquarters 3303 Hillview Avenue Palo Alto, CA 94304 Tel: +1 650-846-1000 +1 800-420-8450 Fax: +1 650-846-1005 TIBCO Software empowers executives, developers, and business users with Fast Data solutions that make the right data available in real time for faster answers, better decisions, and smarter action. Over the past 15 years, thousands of businesses across the globe have relied on TIBCO technology to integrate their applications and ecosystems, analyze their data, and create real- time solutions. Learn how TIBCO turns data—big or small—into differentiation at www.tibco.com. TIBCO Data Virtualization Technical Note Capabilities - Customizing Data Source Properties Project Name Release 2.0 Date April 2018 Primary Author Manny Bhatia Document Owner Tony Young Client Document Location Purpose Demonstrates how capabilities and behavior of a data sources can be changed in TIBCO Data Virtualization.
19
Embed
TDV Technical Note - Capabilities - TIBCO Software...• Using System Adapters and Custom Adapters to alter data source behavior ... • A valid JDBC/ODBC driver and associated JAR
This document is posted to help you gain knowledge. Please leave a comment to let me know what you think about it! Share it to your friends and learn new things together.
Trademarks All brand and product names are trademarks or registered trademarks of their respective holders and are hereby acknowledged. Technologies described herein are either covered by existing patents or patent applications are in progress.
Confidentiality The information in this document is subject to change without notice. This document contains information that is confidential and proprietary to TIBCO Software Inc. and its affiliates and may not be copied, published, or disclosed to others, or used for any purposes other than review, without written authorization of an officer of TIBCO Software Inc. Submission of this document does not represent a commitment to implement any portion of this specification in the products of the submitters.
Content Warranty The information in this document is subject to change without notice. THIS DOCUMENT IS PROVIDED "AS IS" AND TIBCO MAKES NO WARRANTY, EXPRESS, IMPLIED, OR STATUTORY, INCLUDING BUT NOT LIMITED TO ALL
WARRANTIES OF MERCHANTABILITY OR FITNESS FOR A PARTICULAR PURPOSE. TIBCO Software Inc. shall not be liable for errors contained herein or for incidental or consequential damages in connection with the furnishing, performance or use of this material.
Export This document and related technical data, are subject to U.S. export control laws, including without limitation the U.S. Export Administration Act and its associated regulations, and may be subject to export or import regulations of other countries. You agree not to export or re-export this document in any form in violation of the applicable export or import laws of the United States or any foreign jurisdiction.
For more information, please contact:
TIBCO Software Inc. 3303 Hillview Avenue Palo Alto, CA 94304
1.1 Purpose The purpose of this document is to understand how capabilities and behavior of a data sources can be changed in TIBCO Data Virtualization (TDV). The need to alter the default behavior may be either to implement a specific use case, for example, where case sensitivity needs to be altered, or to add new functionality to a data source capability.
This document discusses topics regarding:
• TDV Query Engine process
• TSV Query Engine use of capabilities files.
• Various Sections in TSV Capabilities files
• Process of modification of capabilities files.
• Effects of modification of the capabilities files
• Effects of patches and upgrades to capabilities files
• Using System Adapters and Custom Adapters to alter data source behavior
• Best practices to modify data source behavior
1.2 Audience This document is intended to provide guidance to the following users:
TDV Query Optimizer is the brain behind the TDV Server with its main purpose to analyze, rewrite and execute queries on the underlying data sources. The TDV query optimizer includes a Rules Based and Cost based optimizer, which once invoked by an incoming query performs the following functions:
• Checks the query SQL to validate the syntax
• Identifies resources that are used in the query
o Checks if resources are valid
o Checks if resources is cached and cache is valid
• Identifies the data sources used by the resources
o Checks if data source is enabled and
o Checks if statistics have been enabled and they are valid
• In case of a federated query, rewrites the query for each data source with native syntax.
• Submits the query on the data source connection
• Retrieves the result sets from individual data sources
• Aggregates the results returned from disparate data sources and apply required joins, groupings, sorting or transformations.
For TDV query optimizer to perform the above functions, it needs to know capabilities and behavior of the data sources. This is accomplished by reading capabilities files for each data source. Following sections discuss these capabilities files, how TDV server utilizes these capabilities file and finally, how data source behavior can be modified through the capabilities files.
Before a data source can be used in TDV server, it must be set up and introspected. Data source setup and introspection is described in detail within the TDV User’s Guide. In summary, setting up a relational data source in TDV requires the following steps before it can be used:
• A valid JDBC/ODBC driver and associated JAR files. Some drivers for data sources such as Oracle, Sybase and MySQL come pre-packaged with TDV, while others such as DB2, SQL Server need to be installed.
• Capabilities file, which is a text file that tells the TDV server various capabilities of the data source and the features supported by the data source.
• In the case when a relational data source is not natively supported in TDV and data source does not show up in Data Source creation dialog box, a custom data source may be created based on a generic data source
adapter as long as it supports JDBC and has a JDBC driver.
3.1 What is a Capabilities File? Capabilities files are text files, and as the name suggests they describe the capabilities of a specific data source. These capabilities files determine how a data source will behave during introspection and query execution. We discuss the workings of capabilities file in the following sections.
Every data source used by TDV needs to have its own capabilities file. For example, on a Windows server, capabilities file for Oracle data sources would be located under the C:/Program Files/TIBCO/TDV Server 7.0/apps/dlm/cis_ds_oracle/conf folder, where C:/Program Files/TIBCO/TDV Server 7.0 is the TDV install folder.
Depending on number of data source versions supported by TDV, there could be multiple capabilities files under the ‘conf’ folder. Further, if patches have been applied to TDV server and capabilities files have been modified, multiple versions of these files would be present in the folder. Each of these files follow a naming standard that includes <data source driver version> and <TDV patch number>. The figure below shows a screen shot of capabilities file for Oracle
The above example shows the following details about capabilities:
• Base Oracle capabilities file (oracle-<patch version>.capabilities).
• Capabilities files available for Oracle thin (JDBC) driver for Oracle versions 8i, 91, 10g, 11g and 12c. (oracle_<oracle version>_thin_driver.oracle-<patch version>capabilities)
• Multiple versions of capabilities files for each patch installed on the system (Patch 7.0.5.00.05; 7.0.6.00.00).
• XML version of driver configuration files for each Oracle version (oracle_<oracle version>_thin_driver_config_def.xml).
3.2 Sample Contents of a Capabilities File As noted above, entries in capabilities file determine the behavior of the data source during introspection and also during query execution. This section discusses the contents of capabilities file and how each section of capabilities file is used. For simplicity purposes, we use Oracle data source related capabilities files.
A capabilities file is divided into logical sections, each focusing on certain behavior. Depending on type of data source, these sections may be unnecessary. Examples of these sections are as follows (Note: All capabilities mapping shown are partial sets):
3.2.1 Driver Capabilities As name states, the entries in this section are related to the JDBC/ODBC driver capabilities. Driver properties such as URL pattern, Class name etc. are listed here and when a TDV data source is created, the data source properties are read from this section
Depending on the type of data source, the URL pattern may be different. The table below shows some of the patterns used in TDV while connecting using a JDBC Driver
3.2.2 Introspection Capabilities This section has entries that the TDV server will use during introspection and re-introspection of a data source. Values in this section will tell the TDV server if the data source supports catalogs and schemas, and if to introspect tables, procedures, primary keys, indexes etc.
3.2.3 JDBC Data Source Capabilities Entries in this section tell the TDV query optimizer as how the JDBC driver is expecting the SQL statement, what SQL syntax is supported.
3.2.7 Supported Function Patterns This section contains list of functions that are supported by the data source. Along with the function signature, its usage in TDV is also shown. All functions may have variations on usage and thus multiple occurrences of a same function with different data type may be seen.
AVG(@char).distinct: AVG(DISTINCT $1)
CONCAT(~number,~number): (CASE WHEN ($1 IS NULL) OR ($2 IS NULL) THEN NULL ELSE
3.2.8 Operator Capabilities This section lists all operators supported by the data source.
@char||@char: (CASE WHEN ($1 IS NULL) OR ($2 IS NUL L) THEN NULL ELSE ($1 || $2) END)
@char||@varchar: (CASE WHEN ($1 IS NULL) OR ($2 IS NULL) THEN NULL ELSE ($1 || $2) END)
@timestamp||@varchar: (CASE WHEN ($1 IS NULL) OR ($ 2 IS NULL) THEN NULL ELSE ($1 || $2) END)
@timestamp||@null: NULL
ABS(@char): ABS($1)
ABS(@varchar): ABS($1)@null/@char: ($1 / $2)
@varchar/@varchar: ($1 / $2)
@varchar/~number: ($1 / $2)
@varchar/@null: ($1 / $2)
3.3 How Capabilities Files Are Used At server start up, the TDV server loads all capabilities files into memory. As noted above, there might be multiple versions of capabilities files. The order in which these files are used is very important to the behavior of the TDV query engine. The general rule governing the usage Capabilities file by the TDV query engine is as follows (For example purposes, Oracle 10g data source capabilities files shown in figure above will be used).
• TDV will read the base capabilities file which follows the pattern <data source name>.capabilities: oracle.capabilities
• Next, TDV will read the driver specific capabilities file and overlay the contents of driver specific file over the base data source file loaded in step 1. This file follows a pattern <data source name>_<data source version>_<driver type/version>.<data source type>.capabilities: oracle_10g_thin_driver.oracle.capabilities
• Finally, TDV will read the capabilities file with the highest patch level for the specific data source and overwrite the contents of created by step 2. This file usually follow a pattern <data source name>_<data source
• Starting way back in TDV version 4.5, the concept of adapters was introduced that allows capabilities to be updated via an XML file for each data source adapter rather than through these capabilities files. Therefore, as a last step in this process the XML files will be loaded. Adapters are discussed in following sections.
3.4 Modifying Default Capabilities of a Data Source The above example of a capabilities files demonstrates that all of the entries are name value pairs; which describe a
property and its associated value. Once loaded by the TDV Server, the query engine can then determine how to rewrite a query using the data source capabilities.
For example, when working with Oracle, the query engine knows from the capabilities file, that the data source supports ORDER BY thereby allowing it to use SORT MERGE algorithm when performing a federated join. Similarly, TDV query engine can also identify through the capabilities file, that Oracle supports functions such as RTRIM, TO_DATE, DECODE etc and can push them down.
On the other hand, POSITION function in TDV is translated to INSTRING function of Oracle and TDV query engine can rewrite the POSITION function to INSTRING based on the pattern in capabilities file.
By adding or modifying the Capabilities file, new functionality can be added to a data source or existing behavior can be modified. For example, if a new data type VARCHAR2 needs to be added in Oracle capabilities file, the following line can be added (in Native-to-TDV data type mapping section)
jdbcds.datatype.varchar2: varchar
The above line tells the TDV server that Oracle VARCHAR2 data type is equivalent to VARCHAR in TDV.
Occasionally there is a need to expose a new function capability in a data source. This may be required to push the SQL with a function to the data source. Adding a new data source functionality capability requires multiple steps as
follows:
• A function with the same name and signature as data source function is created in TDV using TDV Studio.
This function may be a placeholder with a valid signature or it could be a fully implemented SQL script procedure.
If the function is a placeholder, only the data source that has the functional capability may use the new functionality and all other data sources will return an error. On the other if the same function is implemented in TDV, when invoked within a SQL, Query Engine will decide if the function can be pushed to the data source. DECODE function is an example of such function where it has been implemented in TDV but is also a built in function in Oracle.
• The created function must be enabled as a global function. This may be done from Administration -> Custom Functions menu.
NOTE: User must be part of an ADMIN group to create a global function.
• The capabilities file must be changed to show the usage. Below is an example of DECODE function in Oracle
NOTE: A TDV Server restart is required for new/modified capabilities to take effect
3.5 Effect of Patches on Capabilities File In the previous two sections, it was discussed that how we can modify the capabilities and how the TDV server uses the capabilities file. Occasionally, patches are installed on the TDV server to provide new functionality or for bug fixes. These patches may sometimes include new capabilities files for one or more data sources.
As the TDV server loads the latest patched capabilities file, any custom modifications done to the existing capabilities files may be over written once a patch installation creates as new capabilities file version. This becomes a maintenance nightmare if TDV administrators have to manually manage the changes in the capabilities files.
3.6 Data Source Adapters – System and Custom Adapte rs Starting way back in TDV version 4.5, TIBCO introduced a concept of Data Source Adapters. Data source adapters provide an easy interface to manage data source customizations. Using the adapters, default capabilities of a relational data sources can be modified without the need to modify the capabilities file as discussed previously. Custom adapters allow similar modifications when required changes to capabilities are limited.
Details on creating and managing adapters are discussed in the “Working with Data Sources” section of the TDV User’s Guide. For purpose of this document, both System and Custom adapters are created under <TDV_HOME>/conf/adapters folder. The system subfolder contains folders for each supported data source version in
TDV with a custom XML values file outlining the capabilities of the data source driver. The example below shows the
structure for Oracle 12c Thin JDBC drivers.
The format of the XML file in this folder is as follows. By default usually the whole file is commented out using XML
Syntax which indicates no overrides are enabled in the XML file and TDV should use the default values from the capabilities files instead. However, here you can see that some overrides have been made into the Oracle 12c
At TDV server startup, this data source XML file is loaded after the base data source capabilities and patched version capabilities files and it overwrites the contents of the capabilities files.
If any further changes are required to the capabilities, either the required sections may be un-commented or new changes can be added in a valid XML syntax. The example below of modified files shows how a capability showing
The default XML file does not contain all capabilities of a specified data source and thus to make required changes, it may be necessary to get the correct XML code from TIBCO Support. A field-implemented utility is also available, which converts the text capabilities file to its equivalent XML format.
NOTE: TDV server should be restarted for the capabilities changes to take effect.
3.7 Best Practices Regarding Modification of Data S ource Capabilities Allowing modification of data source capabilities in the field provides flexibility and adds tremendous advantage to TDV, but it also adds maintenance complexity. Some issues that come up are
• As changes to capabilities file effects the data source and all connections to that data sources, thus it can
have detrimental effect on other application if changes are not properly tested.
• If the changes to capabilities files are not properly documented and communicated, troubleshooting may become difficult.
• If changes to capabilities file enable new functionality, the field team or customer may not communicate it back to TIBCO, thereby making the install out of sync with the core product.
• Any patches applied to a TDV instance may create a new capabilities file thereby overwriting the changes
Therefore, TIBCO Professional Services best practices strongly discourage making changes to the capabilities file
unless absolutely necessary. If the changes are required, they should be communicated to TIBCO Support for logging and general testing purpose.
On the other hand, if changes are required to the capabilities, XML Capabilities file under System or Custom adapters.
Depending on type of modification, either System or Custom adapters should be used. For adding new functionality
that will enhance the data source, like DECODE in Oracle, TIBCO Professional Services best practice recommends modifying the System adapters. On the other hand, if data source behavior needs to be changed, as with case
sensitivity or trailing spaces settings, then creation of custom adapters are recommended.
Finally, some other advantages of using the adapter capabilities files are:
• Subsequent patches applied to the TDV instances do not overwrite the changes made to the adapter XML
files.
• Changes to the adapters capabilities files ARE carried over during Export or Import of a data source to a new instance. Built-in scripts pkg_export/pkg_import commands include the adapter changes as part of export or import.