Carlton Doe and his session – aka the warm-up band for Mark Scranton

Carlton Doe and his session –

akathe warm-up band for

Mark Scranton

Moving from IBM IDS 7.x to IBM IDS 9.x

Carlton Doe

Technical Sales Manager,

Dallas, TX

IBM

3

ISBN 0-13-605296-7 ISBN 0-13-080533-5

?

Who I Am 12 + years of experience as DBA and engine admin mainly in retail

environments Co-founded, presided over, and former member of the Board of

Directors of the International Informix Users Group (IIUG) Written two Informix Press books:

4

What I’m Going to CoverThe Basics

IDS Futures What is IBM IDS and Foundation? Things to Know Before Moving

Reserved Words ONCONFIG Parameters How Do I Move?

Dynamic Logical Log Files Configurable Lock Modes SQL Statement Cache Raw / Standard Tables Changes to “Explain Mode” Fuzzy Checkpoints IBM Informix MaxConnect

5

Smart Large Objects and Dbspaces Simple/Complex Datatypes and Casting User Defined Routines and Functions Collections DataBlades and the Blade Manager HDR / Enterprise Replication – A Snapshot Java in the Engine

What I’m Going to CoverThe Good Stuff

6

IDS FuturesIDS Futures

7

Projected IDS Roadmap

IBM will deliver new releases of IDS about every 18 months

9.6

2003 2004 2005 2006

9.x

In PlanningAccepting Feature Requests

Projected 4Q 2004 Delivery

In Development Features Frozen

Planned 2Q 2003 Delivery

9.4 9.5

In Concept

Projected 2Q 2006 Delivery

8

Development is focused on providing a release of IDS 9 of interest to all IDS customers.

OLTP Features are the #1 Priority

Scalability Security SQL Compatibility Performance Ease of Upgrade And More!

IDS 9.40 Features

9

What can you expect down the road? Further Integration with IBM & DM Products Focus on Security Focus on SQL Compatibility Focus on Performance Leverage the SMART Initiative Support for the Latest Hardware

Platforms Operating Systems Structured Storage – NAS & SAN

IDS 9.50 & 9.60

10

Platform Directions Industry Consolidation, majority of IDS sales are on 5

platforms: Tier 1 Operating Systems: AIX, Solaris, HPUX, Windows, Linux Tier 1 Platforms: Sparc, Power, PA-RISC, Intel, IA64 Planned Continuing support (Planned for IDS 9.40) for Tru 64, Irix, Caldera

OpenUnix + Linux Kernel Personality (LKP)

Future is 64 bit. Many vendors are slowly moving towards IA64

Affected operating systems include: Windows, Linux, HPUX Whenever possible move to 64 bit servers, rather than 32 bit Most applications will run unchanged against a 64 bit server

Some will not be 64 bit clean -> thorough testing prior to deployment is recommended

Full exploitation of 64 bit server may require recompiling and/or modifying application

11

Downlevel Operating System VersionsYou need to move to newer OS levels and/or 64 bit servers

(recommended if available) Solaris 2.5.1 Solaris 2.7 (64 bit) AIX 4.3.2 and below HPUX 10.x Tru 64 4.x Dynix 4.4.x and below Windows NT Linux 2.2 Kernel HPUX 11 32 bit Caldera SCO OpenServer & UnixWare

12

What’s the DifferenceBetween IDS and

Foundation?

What’s the DifferenceBetween IDS and

Foundation?

13

IBM Informix Dynamic Server7.x

IDS - UDO9.1x

What Is IBM IDS and Foundation?

Parallelism built-in Parallel Data Query SQL92 Entry Level Enterprise Replication Many, many more...

SQL 3 Support DataBlade Support DataBlade Developer Kit User Defined Routines User Defined Datatypes User Defined Indexing R-Tree Indexing Extended B-Tree Support Row Types Collections

(sets, multiset, lists) Inheritance Polymorphism Partitioning with new data types

14

IBM Informix Dynamic Server7.x

IDS - UDO9.1x

+ =

IBM IDS 9.2 +IBM IDS 9.2 +

What Is IDS and Foundation?

15

Then What Is Foundation?

WebDataBlade

ExcaliburText

DataBlade

ObjectTranslator

OfficeConnect

J/Foundation

IBM Informix Dynamic ServerIBM Informix Dynamic Server

16

The New Face of the Foundation Family:

IBM IDS with J/Foundation New product -- IDS with the Hot-Spot JVM

Internet Foundation – has been discontinued

Financial Foundation for Capital Markets TimeSeries, NAG DataBlade Modules, TimeSeries Real Time Loader,

Office Connect, Object Translator

Law Enforcement Foundation Visionics, fingerprint, and other biometric-oriented DataBlade

Modules

IBM Informix Spatial DataBlade Module isIBM Informix Spatial DataBlade Module isfreely available to IDS 9.3 customers!!freely available to IDS 9.3 customers!!

17

Which Version Is Loaded??

(IBM IDS)Odra: onstat -

Informix Dynamic Server Version 9.30.UC1G1

(Foundation)Ebro: onstat -

Informix Dynamic Server Version 9.30.UC1G1

(MSGPATH)Wed Oct 4 07:49:52 200007:49:52 Booting Language <builtin> from module <>07:49:52 Loading Module <BUILTINNULL>07:49:57 Informix Dynamic Server Version 9.30.UC1G1 Software Serial Number

AAB#J50070507:50:12 Informix Dynamic Server Initialized -- Shared Memory Initialized07:50:15 Physical Recovery Started07:50:25 Physical Recovery Completed: 0 pages restored

18

Which Version Is Loaded??

Ebro: cd /usr/informix/9_3/extend

Ebro: ls -l

drwxr-xr-x 4 informix informix 1024 Jul 19 08:08 ETX.1.30.UC5

drwxr-xr-x 4 informix informix 1024 Jul 19 08:07 TXT.1.10.UC5

drwxr-xr-x 4 informix informix 1024 Jul 19 09:02 VTS.1.20.UC1

drwxr-xr-x 2 informix informix 1024 Jul 19 07:42 ifxbuiltins.1.1

drwxr-xr-x 2 informix informix 1024 Jul 19 07:42 ifxmngr

drwxr-xr-x 2 informix informix 1024 Jul 19 07:42 ifxrltree.2.00

drwxr-xr-x 4 informix informix 1024 Jul 19 07:49 krakatoa

drwxr-xr-x 4 informix informix 1024 Jul 19 07:42 LLD.1.20.UC2

drwxr-xr-x 4 informix informix 1024 Aug 22 13:36 WEB.4.10.UC1

19

Things to knowbefore you moveThings to know

before you move

20

Things to Know

You can not directly move from < 7.x versions IBM STRONGLY suggests that you first move to the latest

available 7.x version then move to 9.x While you may want to export/import your data, in-place

upgrades have been quite successful -- just remember to wear your belt and suspenders

HP-UX 11.0 - RUN_AS_ROOT.server script is wrongLinks created for one library goes into /usr/lib instead of

/usr/lib/pa20_64; if left alone, engine will not come up (“library not found” error). A “defect” has been entered

** Check 9.3 version, may be corrected.**

21

With IBM IDS, CPU and user-defined VPs are run as user “informix” while other VPs are still run as root. This prevents UDRs from having root access

If you have auto-start scripts, you should modify them so they “su” to “informix” before starting the engine

Install engine after client products for “finderr” to work. The order is:

tools network engine

Things to Know

22

Precision change in float / smallfloat to decimal conversion smallfloat 8 -> 9float 16 -> 17

Default behavior of “create index” is now to create detached indexes. 7.x attached indexes are supported by the upgrade process

Use the “DEFAULT_ATTACH” variable to temporarily mimic 7.x behavior, it will be discontinued however in a future release

9.4 Feature: LVARCHAR increases 2K to 31KOptional parameter added to lvarchar data-type declarations:

LVARCHAR (n) where 0 < n < 32k-2. Current syntax still supported, but old 2k limit holds for that syntax.

Things to Know

23

If using IBM Informix Enterprise Gateway with DRDA version 7.31.UC1 with IBM IDS 7.x, you must get the DRDA patch to avoid defect #132619

HADR has changed. DRAUTO is no longer supported so after a HADR failure condition is declared, the secondary server goes to read-only mode

You must manually convert the secondary server to “standard” mode “onmode -d standard”

No changes to DRINTERVAL, DRTIMEOUT, DRLOSTFOUND

Things to Know

24

9.2 / 9.3 ER and HPL – does not support named and other complex datatypes, only distinct and opaque types

9.4 feature: new functionality! see ER/HDR portion of the presentation.

Test dbexport / dbimport to see if more complex named types work

Shut down HADR / ER before upgrading

Things to Know

25

System catalog changes - columns added, moved, data types changed. New tables added and some dropped. Sysindexes and systables now a view rather than tables

Long identifiers!!The 8/18 rule is dead (the crowd cheers!!)user ids - 32 characters identifiers - 128 characters

table name, server name, database name, column name, index name, synonym name, constraint name, procedure name, dbspace name, blobspace name, optical cluster name, trigger name, type name, routine language name, access method name, operator class name, trace message class name, trace message name, procedure variable names

Things to Know

26

9.4 Feature: File size limits for utilities have been increased from 2GB to 17 billion GB

Utilities Affected:Oncheck, Onload, Onlog, Onmode, Onmonitor, Onparams, Onspaces,

Onstat, Onunload, DB-Access, Dbexport, Dbimport, Dbload, Dbschema, Onpload, Ontape

Also updating Streams API for > 2GB support

9.4 Feature: Chunk size limit changes from 2 GB to 4TB 9.4 Feature: Maximum instance chunks changes to

32,767

Things to Know

Individual instance size now 128 petabytes

27

128 petabytes!Just how BIG is 128 petabytes? At a load rate of 10 gigabytes an hour it would take well

over a 1,000 years to fill a 128 petabyte database.

At the current price for high-end structured storage it would cost $3-4 billion to buy 128 petabytes of storage.

The US Library of Congress contains an estimated 10 terabytes of printed material. 128 petabytes is equal to 12,800 Libraries of Congress.

28

9.4 Feature: # of columns in a Functional Index Old Limit 16 columns New Limit 341 Columns (Java, SPL) New Limit 102 (C)

9.4 Feature: DBServerAliases Old Limit 10 New Limit 32

9.4 Feature: Size of a Shared Memory Dump Old Limit 2 GB limit New Limit 17 Billion GB

Things to Know

29

9.4 Feature: Removed IDS libraries from /usr/lib

Each user connection uses about 5% more shared memory. Watch your shared memory allocations - both instance and operating system This can be offset by using IBM Informix MaxConnect

Make sure you drop and rebuild your distributions after upgrading!

Things to Know

30

Things to Know: New Reserved Words

CACHE COSTFUNC ITEM SELCONST INNER JOIN LEFT

LOCKS RETAIN RAW STANDARD AVOID_EXECUTE USE_SUBQF AVOID_SUBQF

31

Things to Know: One Example of New SQL Functionality9.4 Feature: ORDER BY values not in select list

Most databases implement the ability to order a list based on values not returned by the select. Solution: Implement support for this SQL syntax:

SELECT d.dept_num FROM dept d, employees e WHERE d.dept_num = e.dept_num GROUP BY d.dept_num ORDER BY avg(e.salary);

32

Things to Know:New ONCONFIG Parameters

ALLOW_NEWLINE BLOCKTIMEOUT DD_HASHMAX DD_HASHSIZE DS_HASHSIZE DS_POOLSIZE PC_HASHSIZE PC_POOLSIZE SBSPACENAME SBSPACETEMP SYSSBSPACENAME STMT_CACHE STMT_CACHE_SIZE

STMT_CACHE_HITS STMT_CACHE_NOLIMIT STMT_CACHE_NUMPOOL VPCLASS JDKVERSION JVPHOME JVPLOGFILE JVPPROPFILE JVPJAVAVM JVPJAVAHOME JVPJAVALIB JVPCLASSPATH JVMTHREAD

33

Things to Know:New ONCONFIG ParametersVPCLASS

Enables you to designate and create uniquely named, user-defined, or system classes of VPs. User-defined VPs (UDVPs) have the same functional power as CPU VPs

User-defined VPs should be created to execute user- defined routines and/or DataBlades

Some DataBlades (Verity, Excalibur Text) require their own VP as does the JVM

User-defined VPs can be added/dropped on the fly with the “onmode -p class_name” command

34

Things to Know:New ONCONFIG Parameters

VPCLASSSyntax tree:

VPCLASS name,num=X,[max=X,aff=(X-Y),noyield,noage]

VP name is not case sensitive, options are order independent, no whitespace, 128 char max/entry

Must declare multiple JVPs to execute Java UDRs in parallel (Unix)

If using “noyield”, only declare “1” vp since the UDR will execute serially, causing others to queuefor their turn

35

Things to Know:New ONCONFIG ParametersUse the VPCLASS parameter to replace the CPU and AIO ONCONFIG parameters

VPCLASS cpu,num=3,max=10,noage

Comment out AFF_SPROC, AFF_NPROCS, NOAGE, NUMCPUVPS, SINGLECPUVP parameters

VPCLASS aio,num=5,max=10

Comment out NUMAIOVPS parameter

36

ON-Archive. Use either the ontape or ON-Bar backup utility instead.

DB/Cockpit. Use ISA instead. Informix-DBA. Use Server Studio Java Edition by

AGS instead. ISM graphical user interface. The command line

provides the same functionality.

Things to Know: Discontinued Support

37

Things to Know:How do I Move?

1. Stop database processing and force a change to a new logical log

onmode -sy; onmode -l; onmode -c; onmode -ky;

2. Back up the instance(s) and critical configuration files in $INFORMIXDIR

etc/ /aaodir/adtcfg$ONCONFIG (s)** /dbssodir/adtmasks ONCONFIG.stdsm_versions sqlhosts**tctermcaptermcap

38

3. Restart the instance(s) and put them into single-user mode to ensure all open transactions are properly rolled back

oninit -sv

4. Verify data and index integrity with the oncheck utility (-cr, -ce -cc, -c [ I / D ] db_name)

5. Shut the instance(s) down again and create level 0 backup(s) and/or dbexports – make sure logical logs are backed up and clear!!

6. Install the new IBM IDS / Foundation software and change relevant scripts, global parameters, config files


39

/usr/informix

7_3

9_3

scripts

disks (also tapes)

isa

logs

config_files

max_conn

koetari -- symbolic links

odra -- symbolic links

(current logs) /old_logs

Environment Variables:

INFORMIXDIRONCONFIGINFORMIXSQLHOSTS

My Recommended Directory Structure

40

7. Change ALARMPROGRAM to [ null | no_log.sh ] if using On-Bar or LTAPEDEV to /dev/null

8. Install any DataBlades

9. Restart each instance into quiescent mode and monitor the MSG_PATH file. Sysmaster and reserved pages conversion is automatic (see $INFORMIXDIR/etc/dummyupds7x.sql). When completed, a notice is written to MSG_PATH. If upgrading from 9.2 to 9.3, watch for sysutils and sysmaster db build successful messages in MSG_PATH before doing anything else!

10. Verify data integrity with oncheck commands


41

11. Execute an “update statistics low drop distributions” command on each database in the instance

12. Execute an “update statistics high” command on the sysmaster database

13. Execute an “update statistics medium distributions only” command on each database in the instance.

14. Change LTAPEDEV / ALARMPROGRAM back to its original value


42

15. Create a level 0 backup.

16. Perform system tests to check performance. Execute queries that will stress indexes to see if the optimizer properly uses them as well as their performance.

17. Let the users back in

If there are any problems -- RESTORE from tape


43

Dynamic Logical LogsDynamic Logical Logs

44

Dynamic Logical Logs:Introduction

New feature in IBM IDS 9.3 – automatically create, insert, and activate logical logs when needed!!!

Does not eliminate long transactions, just the server hangs that can occur especially on startup.

Required: Ability to add log on the fly Insert log immediately after current logical log Bring the log into active mode without a backup of rootdbs

/ critical Dbspace(s)

45


In the pre-9.3 engine: Could only be added when in quiescent mode Always took the first available slot in the list Required a backup of the rootdbs (level 0) to become active

46


In IBM IDS 9.3: LOGSMAX disappears from $ONCONFIG Logs can be added while instance is processing

transactions if so configured Logs can be added after current log to avoid running

into log with "begin work" statement Newly added logs are immediately available Need to use DYNAMIC_LOGS $ONCONFIG

parameter

47

Dynamic Logical Logs:How do you add a log?

onparams –a [ -d dbspace ] [ -s size ] [ -i ]

48

Dynamic Logical Logs:How do you add a log?

49

Dynamic Logical Logs:Server actions

The server itself will attempt to add a log file if two conditions are true:

The next active log file contains an open transaction DYNAMIC_LOGS is set to 2 (the default)

The server checks for those conditions at two important points in the code:

Immediately after a log switch Beginning of the last phase of logical recovery (Transaction Cleanup)

50

Dynamic Logical Logs:Server Actions

What Dbspaces are used for automatic log allocation?

1 The dbspace that contains the newest log files. (If this dbspace is full, the engine searches other dbspaces.)

2 Mirrored dbspace that contains log files (but excluding the root dbspace)

3 All dbspaces that already contain log files (excluding the root dbspace)

4 The dbspace that contains the physical log

5 The root dbspace

6 Any mirrored dbspace

7 Any dbspace

51

Dynamic Logical Logs:DYNAMIC_LOGS parameter

Values for the DYNAMIC_LOGS parameter:

2 (default) the server is authorized to add log files automatically to avoid long transaction hangs

1 if a log file is needed, the server waits for the admin to add one manually

0 feature is turned off, mimics < 9.3 behavior

52

Dynamic Logical Logs:DYNAMIC_LOGS parameterWhy set DYNAMIC_LOGS=1?

You want to make use of the feature in order to avoid long transaction hangs, but you don’t want to give up any control over log file location or size.

Remember that once a log file is added to a non-critical Dbspace, that space becomes critical.

53

Dynamic Logical Logs:DYNAMIC_LOGS parameter

Why set DYNAMIC_LOGS=0?

Hmm, tough question. We don’t know.

Adding a log file to a Dbspace forces your next archive of that Dbspace (and the ROOT Dbspace) to be a level 0. If you’d rather risk a server hang than risk having to occasionally tweak your archive schedule, turn the feature off.

If the server does hang due to a long transaction rollback, simply set DYNAMIC_LOGS to 1 or 2 and bounce the server. Now you may have to improvise with your archive schedule, but it’s better than prolonging a down system.

54

Dynamic Logical Logs:DYNAMIC_LOGS parameterDropping logical logs in 9.3 is similar to <9.3

Never-used log files (A) drop immediately Used log files are marked “Deleted”. Once all

spaces have been archived they will automatically drop

If system has never been archived, the new drop restriction isn’t necessary, and is not applied

NEW -- Can be done while instance is on-line!!!

55

Configurable Lock ModeConfigurable Lock Mode

56

Configurable Lock Mode:Introduction

An extension of the "lock mode [ row | page ]" syntax in "create table" command

Allows users / admins to set the lock mode on newly created tables without having to explicitly set it in the table creation statement.

Uses either an environment variable or an $ONCONFIG parameter setting

57

Configurable Lock Mode:How's it set?

IFX_DEF_TABLE_LOCKMODE environment variable

setenv IFX_DEF_TABLE_LOCKMODE row

This variable can also be set in $HOME/.informix or $INFORMIXDIR/etc/informix.rc files

DEF_TABLE_LOCKMODE $ONCONFIG parameter

DEF_TABLE_LOCKMODE row

This sets LOCK MODE to ROW for all tables for all sessions in the server

58

Configurable Lock Mode:Examples

No additional syntax:

{TABLE "carlton".test_table row size = 155 number of columns = 3 index size = 0 }

create table "carlton".test_table ( col1 integer, col2 char(50), col3 varchar(100) ) extent size 32 next size 32 lock mode page;

TBLspace Report for my_nt_test:carlton.test_table Physical Address 200033 Creation date 09/09/2001 15:07:35 TBLspace Flags 901 Page Locking TBLspace contains VARCHARS TBLspace use 4 bit bit-maps Maximum row size 155

59


Explicit "lock mode page" syntax with environmental variable set to "row":

{TABLE "carlton".test_table_3 row size = 155 number of columns = 3 index size = 0 }

create table "carlton".test_table_3 ( col1 integer, col2 char(50), col3 varchar(100) ) extent size 32 next size 32 lock mode page;

TBLspace Report for my_nt_test:carlton.test_table_3 Physical Address 200245 Creation date 09/09/2001 15:18:09 TBLspace Flags 901 Page Locking TBLspace contains VARCHARS TBLspace use 4 bit bit-maps Maximum row size 155

60


With environment variable set to "row" but no additional syntax in "create table" command:{TABLE "carlton".test_table_2 row size = 155 number of columns = 3 index size = 0 }

create table "carlton".test_table_2 ( col1 integer, col2 char(50), col3 varchar(100) ) extent size 32 next size 32 lock mode page, row;

TBLspace Report for my_nt_test:carlton.test_table_2 Physical Address 200034 Creation date 09/09/2001 15:18:09 TBLspace Flags 903 Page Locking Row Locking TBLspace contains VARCHARS TBLspace use 4 bit bit-maps Maximum row size 155

Bug in the

code !!

61

SQL Statement CacheSQL Statement Cache

62

SQL Statement Cache: IntroductionAs implied, prepared and optimized SQL statements are stored in a section of the virtual portion of shared memory. Users executing identical DML SQL statements can pull the pre-parsed query plan and even query data structures from the cache

More complex queries (multiple columns, many filters in the WHERE clause) benefit most from caching

Turned off by default, the STMT_CACHE $ONCONFIG parameter and the “SET STATEMENT CACHE” SQL command control how the caching process works.

Can be overridden with the “onmode -e” command

63

SQL Statement Cache: How It Works

1. Incoming SQL statements are hashed to a value which is compared to other hash values in the SSC section of the virtual portion of shared memory

Caveats: statements only optimized at the PREPARE phase host variable names/placeholders are not included

in the hash algorithm white spaces and case of the statement is significant session-specific parameters (OPTCOMPIND, OPT_GOAL) are

significant and will affect hash value

64

SQL Statement Cache: How It Works

2. If hash value matches an existing value, the cached copy is used. The existing copy is NOT re-optimized

3. If a match does not exist, the statement is evaluated for inclusion in the SSC

Qualifications for inclusion in the SSC: contains DML statements using built-in data types and operators no UDRs/UDFs local database connections only no explicit temporary tables

65

SQL Statement Cache:How It WorksQualifications for inclusion in the SSC (cont):

“select *” statements can be fully expanded to include all columns in the listed tables

is not a statement generated by a stored procedure no embedded subselects in the SELECT statement

66

SQL Statement Cache:Enabling and Sizing

onmode -e {ENABLE|ON|OFF|FLUSH} ENABLEENABLE - Allows statement caching to take place. This does not turn

caching on, but allows it to be turned on

ONON - Turns on statement caching. Caching must first be enabled through the STMT_CACHE parameter or “onmode” command

OFFOFF - Turns off the SSC and immediately disables sharing. Cached statements that are in use remain in the SSC until their use count reaches zero

FLUSHFLUSH - Active statements are marked and subsequently flushed when they are released. When the SSC is turned off, sharing is immediately disabled

67


STMT_CACHE $ONCONFIG parameter:

00 - SQL Statement Cache disabled (default)11 - SSC enabled though sessions must explicitly call

for its use22 - SSC enabled for all sessions unless explicitly denied

on a session-by-session basis

Set the STMT_CACHE environment variable to “0” (off) or “1” (on)

Execute a “set statement cache [ on | off ]”command

68


STMT_CACHE_SIZE $ONCONFIG parameter:determines the size of the SSC in KBs, default = 524 kb

If SSC is full and all statements are active, if additional statements qualify for inclusion the SSC will expand to hold them. As a statement’s user counts drop to zero, the SSC will flush them and reduce back to STMT_CACHE_SIZE

69

SQL Statement Cache:Monitoring and Tuning

“onstat -g cac stmt” displays size of the SSC the cached statements, who’s currently executing a statement, and the number of times it’s been used

The “hash” column is not the statement’s hash value, rather the hash bucket in which the statement resides

To tune, monitor the size of the SSC over time, then resize as appropriate

70


Use the STMT_CACHE_DEBUG environment variable to trace what happens with any given SQL statement

export STMT_CACHE_DEBUG=debug_value:path_to_file

debug_value can be “1” (basic) or “2” (extended)

71

SSC identity (OK:status=1): select * from yoyoSSC link (semantic error): select * from yoyoSSC identity (stmt not found): create table yoyo (col1 int) in mustang_2SSC qualify (invalid stmt type): create table yoyo (col1 int) in mustang_2SSC identity (stmt not found): insert into yoyo values (1)SSC qualify (OK): insert into yoyo values (1)SSC key insert (OK): insert into yoyo values (1)SSC full insert (OK): insert into yoyo values (1)SSC identity (stmt not found): insert into yoyo values (2)SSC qualify (OK): insert into yoyo values (2)SSC key insert (OK): insert into yoyo values (2)SSC full insert (OK): insert into yoyo values (2)SSC identity (OK:status=1): insert into yoyo values (2)SSC link (OK): insert into yoyo values (2)SSC identity (stmt not found): drop table yoyoSSC qualify (invalid stmt type): drop table yoyo


72

New $ONCONFIG parameters: STMT_CACHE_HITS

Number of statement hits before entered into the cache

Values: 0 – statement immediately inserted N – first time statement used, key-only entry made in cache,

from 2 to N counter incremented. After N uses, full statement entered into cache

Want to set to "1" (one) to prevent ad-hoc statements from clogging cache


73

STMT_CACHE_NOLIMITControls insertion of qualified statements into cache after its size

is greater than the STMT_CACHE_SIZE value. While statement cache can grow, this acts as a governor to stop unrestrained growth

Values: 0 – no new statements are inserted after STMT_CACHE_SIZE

reached 1 – unrestricted growth of cache is allowed


74

STMT_CACHE_NUMPOOLSpecifies number of pools to be allocated to the SQL Statement

Cache

Values: 1 (default) to 256

As statement cache increases, number of pools supporting the cache may become a bottleneck. Use the

"onstat –g ssc pool" and "onstat –g spi" commands to monitor usage and spin waits. See pages 4-49 – 4-52 in the Performance Guide for more information.


75

Raw / Standard TablesRaw / Standard Tables

76

Raw / Standard TablesRaw tables have been in XPS for quite some time

and are used in conjunction with its SQL-based HPL or massive static data manipulation

Intended for initial loading and data validation

Similar behavioral attributes regardless of database logging mode, i.e., don't try to use them in transactions.

77

Raw / Standard TablesRaw table attributes

nonlogged permanent table, behaves like table in a nonlogging database

uses light appends, adds rows quickly to end of each table fragments

updates, inserts, and deletes are supported but not logged Can not support indexes, referential constraints, or rollback Can only restore from last physical backup if not updated

since that backup, no MiT restore Fast recovery rolls back incomplete transactions on

STANDARD tables but not on RAW tables Wicked fast to use if done properly!!!

78

Raw / Standard TablesSyntaxcreate raw table xyz

( yada, yada

) in etc., etc.

fragment by whatever;

alter table xyz type [ raw | standard ];

When altered for massive manipulation, make sure you backup the table after converting it back from raw mode!!

79

Changes to "explain mode"Changes to "explain mode"

80

Explain ModeCan now get sqexplain.out information without having to

execute the DML statement!!!!

Two ways to activate this feature: per statement with a directive

select --+explain avoid_execute * from xyz

AVOID_EXECUTE directive works in conjunction with EXPLAIN directive. If EXPLAIN directives is omitted, i.e. if only AVOID_EXECUTE directive is used,the query will not be executed and no explain output will be generated.

81

Explain Mode Set as part of a statement block

set explain on avoid_execute;

After executing this statement, the server writes to the sqexplain.out file without executing any statements until SET EXPLAIN ON/OFF is executed.

82

What Are FuzzyCheckpoints???What Are FuzzyCheckpoints???

83

Checkpoints

IDS 9.2x introduced “fuzzy” operations and checkpoints to increase transactional throughput

Two types of checkpoints:

Full or “sync” Fuzzy

84

Steps of a SyncCheckpoint

1. Engine blocks threads from entering “critical sections” of code2. The page cleaner thread flushes the physical log buffer to log on

disk3. The page cleaner threads flush to disk all modified pages in the

buffer pool (chunk write)4. The checkpoint thread writes a checkpoint record to the logical log

buffer5. The logical log buffer is flushed to the current logical log file on

disk6. The physical log on disk is logically emptied (current entries can be

overwritten)7. The checkpoint thread updates the reserved pages with the

checkpoint record information

85

Good and bad:

logically and physically data, interrupts user activity, expensive (I/O hit), must tune to reduce duration

What Causes Sync Checkpoints to Occur? Physical log becomes 75% full “onmode -c” or “-ky” Administrative actions (adding dbspaces, altering tables) A backup or restore operation using ontape or ON-Bar End of fast recovery or full recovery Reuse of a logical log containing the oldest fuzzy operation not

yet synced to disk LTXHWM reached with fuzzy transactions Through the onmonitor menu

86

Fuzzy CheckpointsRecord “results” of fuzzy operations in the logical log

Goal: reduce or eliminate checkpoint interruptions, eliminate the physical log (over 3 - 4 phases)

Cost: increased logical recovery time

Note: Fuzzy checkpoints are not used in HADR environments

87

Fuzzy Checkpoints

Things to know: Inserts, updates, delete DML statements Currently supports “built-in” data types only (character,

numeric values) Logged databases Not “old” pages. Determined by page timestamp (4

byte, cycles 2 gb -> -2 gb -> 0) No BEFORE images are generated in the physical log During “fuzzy” checkpoints, buffered information on

fuzzy operations is **not** flushed to disk After a fuzzy checkpoint, disks are not physically

consistent

88

Steps of a Fuzzy Checkpoint

1. Engine blocks threads from entering “critical sections” of code

2. The page cleaner threads flush to disk all **non-fuzzy** modified pages in the buffer pool (chunk write) Note: MLRU queues still full

3. A Dirty Page Table (DPT) is constructed containing entries (buffer addresses) for fuzzy operations

4. The DPT is flushed to the current logical log on disk4A. The page cleaner thread flushes the physical and logical log buffers to disk

5. The checkpoint thread updates the reserved pages with the checkpoint information including the Log Sequence Number (LSN).

6. The physical log on disk is logically emptied (current entries can be overwritten)

89

Log Sequence Number (LSN)

A value representing a position in the logical log (log # and log position)

Stored in each dirty buffer heading containing fuzzy operation results along with a newly-stored timestamp

The oldest LSN is tracked in shared memory because it contains the oldest fuzzy operation. Oldest LSN can not be further back than 1 logical log and 2 checkpoints

Logs after the LSN can not be freed for re-use until a checkpoint has occurred

Net Net -- large logical logs are GOOD!!!

90

What Causes Fuzzy Checkpoints to Occur? Checkpoint interval has elapsed Physical log becomes 75% full onmode -c fuzzy Reuse of a logical log containing the last

checkpoint (either fuzzy or sync)

91

Recovery - Sync Checkpoint

1. Physical log data used to return all disk pages to original “synced” state (physical restore)

2. The most recent checkpoint (sync) record is located in the logical log files

3. All subsequent logical log records are rolled forward

4. Uncommitted transactions are rolled back

92

Recovery - Fuzzy Checkpoint

1. Physical restore proceeds as normal however disk pages are not physically consistent since there are modified pages that were not logged in the physical log

2. Logical recovery begins in two phases: Phase A -- reapply DPT records Phase B -- post checkpoint records

93


Phase A - Logical Recovery:1. Locate the oldest LSN representing unwritten data2. Selectively apply fuzzy operations from the

DPT/logical log from the oldest remaining logical log up to the most recent checkpoint

Conditional update based on page timestamp - if DPT record is “newer” than page timestamp, then the change is applied

Result: similar to end of physical restore when using sync checkpoints

94


Phase B - Logical Recovery:1. Beginning with last checkpoint, roll forward

subsequent logical log records.

2. Uncommitted transactions are rolled back

Result: similar to the logical restore when using sync checkpoints. Disks are physically and logically consistent to last recorded transaction

95

Fuzzy Checkpoints

Buffer structures, logical log records, and reserved pages are automatically upgraded when migrating to 9.x

Fuzzy checkpoints enabled by default. Set the NOFUZZYCKPT environment variable to true/on/1 to disable them prior to initializing shared memory

Set the TRACEFUZZYCKPT environment variable to trace fuzzy checkpoints. It will affect instance performance but will log in the MSGLOG the number of buffers that remain dirty after a checkpoint is completed. It also lists the location of the old LSN

96

IBM Informix MaxConnectIBM Informix MaxConnect

97

IBM Informix MaxConnectIBM Informix MaxConnect is a middle-ware communications component that removes the overhead of managing user connections from the instance

Helps you get scale your user connections past the effective 2,000 user connection limit without any change or resource impact to the database server

It supports 7.3 and 9.2x+ Unix ports of the engine but accepts user connections from any platform

NO CHANGE to existing applications!!

98

IBM Informix MaxConnect Performance

BaaN IVc World record: 11,445 BRUs on HP N4000 (8CPU, 16GB) and IDS 21% better than Oracle on identical HP system Beat Oracle’s 64-CPU Sun E10000 result 18,650 database connections using 98 transports !!

99

MaxConnect

MaxConnectIDS/Fnd

SQLClients

For large, hardware-partitionable SMP systems

MaxConnect

MaxConnect

SQLClients

Database server system

IDS/Fnd

IBM Informix MaxConnect Architecture

100

IBM Informix MaxConnect Configuration An IBM MaxConnect instance connects to only one DB

server instance Connections are managed through the SQLHOSTS file

with a new connection protocol -- imc Clients change hostname and port address (maybe) to

point to the IBM MaxConnect instance, instance name remains the same

IBM MaxConnect instance has an ADMIN entry, a client-facing entry, and a DB server-facing entry

DB instance has a DBSERVERALIAS and a DBSERVERNAME entry

101

Client SQLHOSTS file:ebro_tcp ontlitcp maxcon_1 1526

MaxConnect SQLHOSTS file:ebro_tcp ontlitcp maxcon_1 1526ebro_db ontliimc cronus 1575ebro_adm onsoctcpmaxconn_1 1600

Server SQLHOSTS file:ebro onipcshm cronus ebro_ph

(DBSERVERNAME)ebro_tcp ontliimc cronus 1575

(DBSERVERALIAS)

IBM Informix MaxConnect Configuration

102

NETTYPE entry changes:

ipcshm,1,20,CPU

# tlitcp,5,200,CPU

tliimc,1,2000,CPU or NET

MaxConnect instance environment variables:IMCONFIG - location of MaxConnect config file -

default is $INFORMIXDIR/etc/IMCconfigIMCSERVER - Client-side MaxConnect instance

nameIMCADMIN - Admin-side MaxConnect instance

name


103

IBM Informix MaxConnect config file contains:

IMCLOG - location of MaxConnect message log

IMCWORKERTHREADS - # of worker threads, range = 1 -> 64

IMCWORKERDELAY - amt of time MaxConnect will wait and pool messages to transmit to DB server default = 0, range = 0 -> 100,000 ms

IMCTRANSPORTS - # of transport connections to DB

server. Default = 2, range = 1 -> 64, ratio = 1/100 users


104

IBM Informix MaxConnect Tuning

Use “onstat -g imc”, the imcadmin utility, or the ISA to monitor and tune IBM Informix MaxConnect

Tune the NETTYPE sessions parameter by monitoring the q-exceed and alloc/max values. May also need to tweak the IFX_NETBUF_PVPOOL_SIZE variable

Tune IMCTRANSPORTS by monitoring blocked or partial writes

Tune IMCWORKERTHREADS by monitoring AvgQlen if > 5, increase number of threads

Tune IMCWORKERDELAY by summing and comparing histograms against AvgQlen

105

Large Objects, SBSpacesand

Dynamic Lock Allocation

Large Objects, SBSpacesand

Dynamic Lock Allocation

106

I/O size configurable by page size

configurable by page size, the data is still stored in database server sized pages, but the I/O is configurable

Buffer pool usage

partition blobs use the buffer pool, blobspace blobs use private buffers

you can choose between light I/O or normal buffering when the smart-large-object is opened

Updating inserts and deletes only ; there are no direct updates

updates are done in place or moved as needed

Data row/blob mapping

one large-object can be referenced by one data row

one large-object can be referenced by one or more sources

Simple Smart

Large Objects and SBSpaces

IBM IDS has 4 types of LOs: Simple - BYTE & TEXT Smart - CLOB & BLOB

107


Access to C/BLOBs is faster due to “portion-level” addressing Default characteristics are easy to override With sufficient resources, C/BLOBs are transactionally recoverable

Simple SmartLoggingMethod

logging dependson the table wherethe large-object isstored

logging can be turned on or off at thelarge-object level

Size 2 GB maximum 4 TB maximum

LockingGranularity

all or nothing can lock portions or all of the object

Reads andwrites

only the wholelarge-object canbe read

all or a portion of the large-object canbe read

Fragmentation all large-objectsfor a table arestored in the sameblobspace

ability to store the large-objects forone table in multiple sbspaces

108

header (reserved) pages

metadata areaUser Data

Size and location ofthe metadata area isconfigurable atsbspace and/or chunk creation

Metadata is alwayslogged regardless ofdatabase or sbspacelogging mode


Smart BLOBs are stored in “smart” BLOBSpaces (sbspace) which contain meta-data as well as user data

109


onspaces -S name -g pageunit -p pathname -o offset-s size [ -m pathname offset ] -Ms mdsize -Mo mdoffset -Df default list -t

-Mo mdoffset offset, in kbs, into the disk partition where metadata will be stored

-Ms mdsize the size, in kbs, of the metadata area for the initial chunk, remainder is user data space -- not recommended though possible. Each SLO needs ~500 bytes of metadata space

-Df default list default specifications for objects stored in the sbspace. Comma separated, in double quotes (“ “)

-t new option in 9.3 for temporary sbspaces to store temporary smart LOBs

110

Tag Values Default DescriptionACCESSTIME ON or

OFFOFF ON – instance tracks access time

of SLOs stored in the sbspace.AVG_LO_SIZE For

WindowsNT:4 to 2^31

For UNIX:2 to 2^31

8 Average size, in kbs, of SLOsstored in the sbspace. Used tocalculate metadata area size if -Msoption is not specified.Error 131 is returned if you run outof metadata area in the sbspace.To allocate additional chunks tothe sbspace that consist ofmetadata area only, use the -Msoption.

BUFFERING ON orOFF

ON ON - instance uses residentportion buffer pool for SLO I/Ooperations.OFF - instance uses light I/Obuffers in the virtual portion(lightweight I/O operations) –PREFERRED !!!!

LOCK_MODE RANGE orBLOB

BLOB RANGE - only a range of bytes inthe SLO is locked.BLOB - the entire SLO is locked.


111

Tag Values Default Description LOGGING ON or

OFF OFF ON - instance logs changes to the

SLOs in the user data. Create a level-0 backup of the sbspace after creating/changing

EXTENT_SIZE 4 to 2^31

16 Size, in kbs, of the first disk allocation for SLO storage when you create the table. Let the system select the EXTENT_SIZE value.

MIN_EXT_SIZE 2 to 2^31

4 Minimum amount of space, in kbs, to allocate for each SLO.

NEXT_SIZE 4 to 2^31

16 Size, in kbs, of the next disk allocation for smart large objects when the initial extent becomes full. Let the system select the NXT_SIZE value.

Create a level 0 backup after creating a sbspace


112

Large Objects and SBSpaces:New stuff

Temporary SBSpaces: Like temporary DBSpaces -- no logging whatsoever occurs Set with the SBSPACETEMP $ONCONFIG parameter

Temporary smart LOBs: To create a temp smartblob, set the LO_CREATE_TEMP flag in

the ifx_lo_specset_flags or mi_lo_specset_flags function. Use mi_lo_copy or ifx_lo_copy to create a permanent smart large object from a temporary smart large object.

If you put a temporary smartblob in a permanent sbspace it gets deleted at the end of session

113

Metadata stealing:

40% of user data area reserved for either meta or user data, is allocated as needed to either portion (10% increments)

If metadata area is filling, messages in MSG_PATH

If metadata fills, and reserve is empty, no more S-BLOBs can be inserted even if user data area has free space

Must add more metadata space

Large Objects and SBSpaces:New stuff


metadata areaUser Data

114


To add a chunk, you can specify whether the chunk will contain only user data, metadata, or both:

onspaces -a sbspace -p pathname -o offset -s size

[-Ms mdsize -Mo mdoffset | -U ][ -m pathname offset ]

To drop the sbspace:onspaces -d sbspacename -f (force)

115

create table products (prod_sku int, prod_desc clob, prod_photo blob) put prod_desc in (sbspace_1, sbspace_2), prod_photo in (sbspace_9, sbspace_10), fragment by expression (prod_sku < 1000 in db_1, prod_sku >= 1000 in db_2);


SLOs can be distributed in round-robin form among sbspaces like “regular” data

Creates a 72 bytedescriptor “handle”as opposed to 56bytes with “simple”LOs

116


IBM recommends that you convert “simple” LOs to their “smart” counterparts as soon as feasible

You can, in many cases, use the "alter table" command to make the change(s):

alter table t_name modify col_name [ clob | blob ]put col_name in (sbspace(s));

117


Monitoring / tuningoncheck -cs -cS / -ps -pS to print/view metadata

-ce / -pe has additional sbspace information

onstat -g smb s | c | h | e | tsmb s sbspace summary informationsmb c sbspace chunk informationsmb fdd LO file descriptor tablesmb lod LO header tablesmb t timing statistics for enabled server,

not well documentedsmb e undocumented

onstat -k has additional columns to document SLO byte range locks

onstat -d has additional chunk-related columns for sbspaces

118

Chunk adjunct partition

Level 1 backup partition

Level 2 backup partition

LO header partition

Sbspace descriptor partition

user-data free-list partition


metadata area


Backing up and restoring SLOs and sbspaces now operates like regular dbspaces -- at the page level. As a result, you can now have granular and truly incremental SLO/sbspace backups.

119

OtherOtherspacesspaces

Sbspace Sbspace headerheader

Metadata Metadata pagespages

Archive Archive descriptor pagesdescriptor pages

sbspace sbspace pagespages


Backup: Metadata LO headers are read and a list made of pages to be backed up All individual page timestamps are compared too since updates can occur

without affecting the LO header timestamp Qualifying data is sent to the backup program through large buffer accesses

Restore: Sbspace header and metadata information is restored Archive desriptor pages contain page addresses and extent sizes for the

sbspace pages that follow. This is held in shared memory during the operation

Sbspace pages are restored to their correct places

120

Dynamic Lock Allocation In IBM IDS, the locking mechanism changes. LOCKS is now a starting allocation (resident portion) If the lock table fills, additional locks are allocated (x 16) minimum allocation: smaller of current lock table or 100,00 (virtual portion) To prevent lock table flooding, range locks will be upgraded to the whole SLO if a single transaction uses > 10% of lock table “Byte range” locks are what lock portions of a SLO. They are sorted lists of rowids, partition #s, and byte address ranges

121

Dynamic Lock Allocation Range locks can be joined or split as needed if ranges to

be locked adjoin, overlap, or become separate

Ranges locks are not stored in the lock hash table, rather an “intent” lock is placed for the LO header. Byte range locks are linked to associated hash intent lock then sorted by byte number

Byte ranges locks allocated if: < 50% of lock table is being used 50% <= lock table <= 70%, whole SLO lock will be attempted, if

unsuccessful, byte lock allocated > 70% whole SLO locks only, existing locks will be upgraded

automatically

122

Complex and User-Defined Data Types


123

OpaqueOpaqueOpaqueOpaque DistinctDistinctDistinctDistinct

Row Data TypeRow Data TypeRow Data TypeRow Data Type

NamedNamedNamedNamed UnnamedUnnamedUnnamedUnnamed

CollectionCollectionCollectionCollection

MultisetMultisetMultisetMultiset ListListListList

SetSetSetSet

User-DefinedUser-DefinedUser-DefinedUser-DefinedComplexComplexComplexComplex

Extended DataExtended DataTypesTypes


BooleanBooleanInt8Int8

Serial8Serial8LvarcharLvarchar



New Built-inNew Built-inTypesTypes


Existing Built-inExisting Built-inTypesTypes

Data TypesData Types


124

Built-in Data Types

New Built-In: int8 (8 bytes) serial8 (8 bytes)

Range is -9,223,372,036,854,775,807 to 9,223,372,036,854,775,807 must add UC to serial8 to ensure uniqueness one serial8 and serial per table

boolean valid values: “t”, “f”, null. case sensitive

lvarchar (variable length character data type; 32k maximum – check your system for on-disk maximum)

125






SetSetSetSet













126

No “alter type” statement, must drop and recreate

Sys Adm is more complexIntuitive

Not simple SQLRefers to a group of elements by a single name

More complexLess coding

DISADVANTAGESADVANTAGES

Complex Data Types:Row

analogous to C structure, come in two “kinds”NAMED - strongly typed, ID’ed by name, has inheritance, used

to build columns and tables

UNNAMED - weakly typed, ID’ed by structure, no inheritance, used to build columns

Can contain built-in, collection, opaque, distinct, another row type data types

Caveat: no serial or serial8

127

Named: create row type name_t (fname char(20), lname char(20));

create row type address_t (street_1 char(20), street_2 char(20), city char(20), state char(2), zip char(9));

create table student(student_id serial, name name_t, address address_t, company char(30));

Unnamed:ROW (a int, b char (10))

Note: is also equal to ROW(x int, y char(10))

create table part (part_id serial,cost decimal,part_dimensions row

(length decimal,width decimal, height decimal, weight decimal));


128

Using Named Row Types in SQL statements:

Insert statement:insert into student values (1234, row (“John”,”Doe”)::name_t, row ("1234 Main Street","", "Anytown","TX","75022")::address_t, "Informix Software")

Select statement:select * from student where name.lname matches "Doe”;

Result set: student_id 1234 name ROW('John ','Doe ') address ROW('1234 Main Street ',' ,'Anytown ','TX','75022 ') company Informix Software

Use of datatypeUse of datatypekeywordkeywordUse of datatypeUse of datatypekeywordkeyword

Cast the row type!Cast the row type!Cast the row type!Cast the row type!

Access data withAccess data with‘‘dot’ notationdot’ notationAccess data withAccess data with‘‘dot’ notationdot’ notation


129


To drop a named row type:

drop row type address_t restrict;

130






SetSetSetSet













131

Complex Data Types:CollectionsGrouping of elements of the same datatype

(char, int), max size = 32 KB

Used when The data is meaningless without the context of the

other members in the collection (e.g., golf scores, to-do list, set of names)

Individual data elements are not likely to be directly queried by position

The maximum number of data elements is less than 32

Can be null

132

Complex Data Types:Collections

Three kinds of collections:

Set - unordered, no duplicates allowed set {“apple”, ”orange”, ”grapefruit”, “plum”}

Multiset - unordered, duplicates allowedmultiset {“apple”, “orange”, “grapefruit”, “apple”, “plum”, “grapefruit”}

List - ordered, duplicates allowedlist {“apple”, “orange”, “grapefruit”, “apple”, “plum”, “grapefruit”}

133


create table class(class_id serial, class_name varchar(60), description lvarchar,

prereqs set(char(20) not null));

Insert syntax is similar to named row types:

insert into class values (300, “Performance and Tuning”, “Covers advanced information on tuning the Informix Dynamic Server”, (SET{“RDD”,”BSQL”}));

Use the “in” keyword to query values in a collectionselect * from class where (“ASQL”) in prereqs;

define xyz char(20)define set_var set(char(20))select prereqs into set_var from class where class_id = 300

foreach del_set_cursor forselect * into xyz from table(set_var)if xyz matches “RDD” then

delete from table(set_var) where current of del_set_cursorend if

end foreach

134


You can not update one element in a collection, you must replace the whole collection:

update class set prereqs = (set{“RDD”,”ASQL”,”BSQL”}) where class_id = 300;

update class set prereqs = set_char where class_id = 300;

135






SetSetSetSet













136

User-Defined Data Types:Distinct

Two user-defined data types (UDTs):Distinct

data type modeled on an existing data type has a unique name to distinguish it from other similar “types” inherits the internal structure from the source type might have operations and casts defined over its

source type

Opaque data type that is unknown to the database server you must define the internal structure, functions,

and operations

137

create distinct type dollar as decimal;create distinct type aus_dollar as decimal;

create table sales ( sku int, sales_date date, us_sales dollar, aus_sales aus_dollar);

insert into sales values (1234, today, 15.0::dollar,0::aus_dollar);insert into sales values (5678, today, 0::dollar, 75.0::aus_dollar);

select sku, (sum(us_sales) + sum(aus_sales)) from sales where sales_date = today group by 1;

error: 674 - routine (plus) can not be resolved


138

Need to create some UDFs that handle the type and value conversion for you:

create function usdlr_to_ausdlr(parm1 dollar) returning aus_dollar specific usd_to_ausd; return (parm1::decimal * 1.8)::aus_dollar;end function;

create function ausdlr_to_usdlr(parm1 aus_dollar) returning dollar specific ausd_to_usd; return (parm1::decimal / 1.8)::dollar;end function;

select sku, (sum(us_sales) + sum(ausdlr_to_usdlr(aus_sales))::dollar) from sales where sales_date = today group by 1;


139

User-Defined Data Types:Opaque

An opaque data type stores a single “value” that cannot be divided into components by the engine.

Implemented as C or Java structures and manipulated by a set of routines written in C or Java

An opaque “value” is stored in its entirety by the engine without any interpretation of the contents or its structure

All access to an opaque type is through functions written by the user. You define the storage size of the data type and input and output routines

140


1. Create the C / Java data structure to represent the internal data structure

2. Write the support functions in C / Java3. Register the opaque data type with the “create opaque type”

statementcreate opaque type type_name ( internallength = length,

alignment = num_bytes);length is in bytes, alignment = 1,2,4,8 bytes (default = 4)

create opaque type type_name ( internallength = variable,maxlen = length);

length - default = 2 KB, max value = 32 kb

4. Register the support functions with the “create function” statement. If in Java, will be stored in sbspace(s)

141

create opaque type my_type(internallength=8, alignment=4);

create function support_in(lvarchar)

returning my_type with (not variant);external name “/funcs/my_type.so”language C

end function;

create implict cast (lvarchar as my_type with support_in);

5. Grant access to the opaque data type and support functions6. Write any user-defined functions needed to support the opaque data

type - input, output, destroy, compare, aggregates, send, receive, etc.

7. Provide any customized secondary-access methods for creating indexes


142

Casts and CastingCasts and Casting

143

Casts allow you to make comparisons between values of different data types or substitute a value of one data type for a value of another data type.

create function ausdlr_to_usdlr(parm1 aus_dollar) returning dollar specific ausd_to_usd; return (parm1::decimal / 1.8)::dollar;end function;

Engine provides a number of “built-in” casts (int to decimal, numeric to char, etc.) for most built-in datatypes

Must create user-defined casts for user-defined types. Must be unique with respect to source and target data types

Can not create casts for collections, LOBs, or unnamed row types

Casts and Casting

144

Casts and Casting

Two kinds of user-defined casts:explicit

. . . where price >= (cast aus_dollar as dollar) . . . .

. . . return (parm1::decimal / 1.8)::dollar;create explicit cast (aus_dollar as dollar with us_dlr_to_us_dlr);

implicitcreate implicit cast (aus_dollar as dollar);create implicit cast (aus_dollar as dollar with aus_dlr_to_us_dlr);

Automatically invoked when:one data type is passed to a user-defined routine whose parameters are of

another data typeexpressions are evaluated that need to operate on two similar data types

145

Casts and Casting – Implicit Casts

select sum(us_sales) + sum(aus_sales) from sales; # 674: Routine (plus) can not be resolved.

create implicit cast (aus_dollar as dollar);select sum(us_sales) + sum(aus_sales) from sales;

(expression) 120.00

drop cast (aus_dollar as dollar);create implicit cast (aus_dollar as dollar with ausdlr_to_usdlr);select sum(us_sales) + sum(aus_sales) from sales;

(expression) 80.00

146

select sum(us_sales) + sum(aus_sales) from sales; # 674: Routine (plus) can not be resolved

create explicit cast (aus_dollar as dollar);select sum(us_sales) + sum(aus_sales) from sales; # 674: Routine (plus) can not be resolved

select sum(us_sales) + sum(aus_sales)::dollar from sales;(expression) 120.00

drop cast (aus_dollar as dollar);create explicit cast (aus_dollar as dollar with ausdlr_to_usdlr);select sum(us_sales) + sum(aus_sales)::dollar from sales;

(expression) 80.00

Casts and Casting – Explicit Casts

147

Previous examples were “straight” casts. You can also create cast “functions” to cast types with dissimilar data structures

1. write the cast function in C / Java / SPL

create function opaque_to_opaque (input_arg my_type_1)returns my_type_2return cast(cast(input_arg as lvarchar) as my_type_2);

end function;

2. register the cast function with the “create function” command

3. register the cast with the “create cast” commandcreate explicit cast (my_type_1 as my_type_2 with opaque_to_opaque);

Casts and Casting

148

User-defined Routines(UDRs)

User-defined Routines(UDRs)

149

UDRs

There are User Defined Functions and Procedures (UDFs and UDPs)

Can be internal or external in nature. Internal are written in SPL, external can be written in C or Java.

If written in Java, compile into a class then .jar file. When registered in the engine, .jar file will be brought into a sbspace then executed when needed by the JVM in the engine.

If written in C, create a shared object library in a directory owned by “informix” with 755 permissions.

150

UDRs UDRs should be concise and precise. Don’t want engine wasting resources plowing though verbose code. The values returned should be relevant from a business perspective (e.g. proper domain)

Make sure you include error handling in the code

**ALWAYS** use a “specific” name, or alias, for every UDR to properly identify each one - used to drop, grant, or revoke permissions as well as to permit function overloading

151

UDRs

Function overloading occurs when two or more functions have the same name but different signatures

signature = UDR name and parameter list

create function plus (in_1 dollar, in_2 aus_dollar) . .create function plus (in_1 aus_dollar, in_2 lira) . . .create function plus (in_1 lira, in_2 aus_dollar) . . .

{Note: these probably should be formalized into user-defined aggregates}

When properly registered, instance will use the data types passed to determine which UDR should be used

152

UDRs create function routine_name (param_list)

{ returns | returning } typename[ specific specific_name ][ with (internal | handlesnulls | [ [not] variant]

| class=“vp_class”) ]external name ‘full_path_name_of_file‘language language [ [not] variant ]

end function;

in the $ONCONFIG file:VPCLASS fince_vp ,num=2 # vps for finance UDRs

oronmode -p +2 fince_vp

USE UDVPs FOR UDRs!!!!

153

UDRs To drop UDRs, must either use full signature

drop function plus(lire, aus_dollar);

or the “specific” name

drop specific function lir_to_ausdlr;

154

UDRs Default behavior of UDRs is single-threaded, can be “parallelized” if the following conditions are met:

only in DSS environments (PDQPRIORITY > 1) "parallelizable" keyword is added to UDR creation

statement external UDRs only (C / Java) that use PDQ thread-

safe SAPI calls (see DataBlade manual for list of SAPI calls)

no complex types in input parameters or return values

multiple fragments must be scanned

155

UDRs Not directly called (singleton execution) not an iterator - called in loops to return multiple

values similar to with resume in SPL

On some UDRs, you can add “selectivity” and “cost” modifiers that will affect optimizer activities and query plans used. Not really familiar with this

156

UDRs

create function bigger(int,int) returns intspecific big_parallelwith(parallelizable,class=“test_vps”) external name“/path/plludr.bld(bigger)” language C;

alter function bigger (int,int) with (add parallelizable, add class=“test_vps”);

Can monitor the parallel execution of UDRs through “set explain out” command the ISA onstat -g mgm onstat -g ses sess_id onstat -g ath

157

DataBlade Modules and the Blade Manager

DataBlade Modules and the Blade Manager

158

DataBlade Modules DataBlade Modules extend the functionality of the engine by

adding new UDTs, casts, interfaces, UDRs, error messages, client code (where necessary) to manipulate the UDTs

Functionality available through SQL, SPL, API calls to external functions

Can be mixed and matched to support any application

While the code is loaded at an instance level, blade registration occurs at a database level. With registration information contained in the database system tables, if you drop the database, blades become unregistered

159

DataBlade Modules

Install the blade in $INFORMIXDIR/extend (tar, cpio, ?)

May have its own “install” script to brand binaries, use serial # / license key from engine install unless third-party requires unique key setup

Use the Blade Manager to manage blade registration graphical version available with NT and ISA command line only on Unix / Linux

Is case sensitive

160

DataBlade Modules BladeManager supports the following commands:

list [database] - print list of registered DataBlade modulesset user [user]set server [server]show modules - print list of DataBlade modules on servershow databases - print list of databases on servershow servers - print list of serversshow client - print DataBlade modules with files on clientregister [module] [database]unregister [module] [database]add client [module] - install DataBlade module's client filesdel client [module] - remove DataBlade module's client filesinfo [module] - print information about a DataBlade moduledel logs - delete all the log filesshow log - prompt to display a logshow last log - display log from last actionhelp, ?bye, quit, exitset confirm on|off

161

DataBlade Modules

mustang>show databaseDatabases on server: agg1 dcs_demo dps_demo ireach kiosk media360 my_test_db visclass20

mustang>list media360DataBlade modules registered in database media360: lld.1.20.tc2 TXT.1.10.TC5 ifxrltree.2.00 ifxbuiltins.1.1 web.4.10.TC1 VTS.1.20.TC1 video.2.00.TC2

162

HDR / Enterprise Replication –A Snapshot of New Features

HDR / Enterprise Replication –A Snapshot of New Features

163

HDR and 9.4 HDR now supports:

Enterprise Replication (co-existence) Logged Smart Blob Spaces Some DataBlades (when associated smart blob

spaces are logged) Supported DataBlades:

• Spatial• Geodetic• Web• Time-Series• Excalibur Text

164

ER – A SnapshotNew Stuff

DataSync performance improvements Spooling improvements Serial column primary key support Changed columns sent with update, partial row update Smart LOBs support No raw table support Distinct / opaque type support but not complex OLD!! 9.4 Feature: Transaction size limit increases from amount of

shared memory to 4 TB 9.4 Feature: support for additional extensible types:

Row Types Lists Sets Multi-Sets

165

DataSync performance improvements include: Support of intra-replicate parallelism

Parallel apply with ordered commits Operations for a given row are serialized

Necessary to prevent ‘delete before insert’

Some tables are always serialized Page level locking or small tables

Out-of-order commits Allows commit order on target to be different than it was on source

No prior cross replicate relationships No prior referential constraint relationships


166

Decreased logical log consumption DataSync now uses buffered logging Acknowledgements are ‘buddy-bunched’ Shipping of ACKs coordinated with log flush, every 5 seconds or 50 Txn

Is automatic and self-regulated CDRM_Monitor thread

Maintains weighted history of lock failures Controls the degree of parallelism allowed Monitors distance before DDRBLOCK Coordinates ACKs with log flush Recalculates Table Statistics, user may need to run update statistics on a

target only system

Does not provide for intra-transaction parallelism (is not PDQ)


167

9.30 ER interoperates with ER on 7.31 or 9.2x (but NOT 7.30)

Limitations 9.30 does not support replicate groups, all groups should be

deleted from 9.2x/7.31 servers before inter-operation begins 9.2x/7.31 do not support replicate sets; no replicate sets should

be defined until all 9.2x/7.31 servers have been converted to 9.30

PTS 128485: must be applied to 9.2x/7.31 server(s) before beginning interoperation


168

ER – A SnapshotResult!!

9.30

9.21

Apply on Source Apply on Target

169

Introduction to Javain the Server

Introduction to Javain the Server

170

EJB

EJB

Java

JVMJVM

Applet JavaBeanJavaBean JavaBeanJavaBean

RMI | Corba | IIOP RMI | Corba | IIOP

Web Server Web Server

Applet JavaBeanJavaBean JavaBeanJavaBean

Java DBDK

JavaBeanJavaBean EJBEJB

Java in the Server J/Foundation is the marriage of the Java and SQL languages

Can write internal or external Java routines that are accessed natively by the engine

171

Java in the Server:How It Works

Java UDRs execute on Java Virtual Processors Java VPs have the same capability as CPU VP

It can process any SQL query

Avoids the inter-process communication overhead for executing Java UDRs

The Java Virtual Machine (JVM) is embedded directly into the VP code

Java runs threads in parallel

Dynamically load Java VPs Increase JVPs to balance application load

172

IBM IDS 9.30 was enhanced to take advantage of JDK 1.3 There are 3 different Java VM’s

Classic VM: jdk1.2.x VM Client HotSpot VM: tuned for user-interaction apps needing short start-

up & smaller memory footprint (JIT less aggressive) HotSpot is from Sun, IBM has other VMs that are not based on HotSpot technology.

Server HotSpot VM: tuned for servers, used in our J/Foundation product

Classic VM was used for IDS 9.21, and is not supported for the 9.30 release. This is in line with the vendor roadmaps for JDK.

Java in the Server:New Stuff

173

Performance improvements in 9.3 J/Foundation include: thread pooling, code optimization of Java UDR invocation, support for HotSpot Server VM

Performance improvements will vary by the application, but internally we’ve seen an improvement of approximately 135% for our test suites.

Java in the Server:New Stuff

174

Java in the Server:Configuration

New ONCONFIG Parameters:

VPCLASS jvp,num=1

JVPJAVAHOME /usr/java

JVPHOME /opt/informix/extend/krakatoa

JVPPROPFILE /opt/informix/extend/krakatoa/.jvpprops

JVPLOGFILE /opt/informix/jvp.log

175

9.21 $ONCONFIG had the settings JDKVERSION 1.2 JVPJAVAVM hpi:jvm:java:net:math:zip:jpeg:dcpr

9.30 $ONCONFG will need the settings JDKVERSION 1.3 JVPJAVAVM hpi:server:verify:java:net:zip:jpeg

Look at examples in onconfig.std

Java in the Server:Configuration

176

Java in the Server:JDBC Driver Information We provide a “Type 4” JDBC driver

Type 1 JDBC-ODBC bridge provides JDBC access via most ODBC drivers

Type 2 A native API and partly Java driver that converts JDBC calls into calls on the client API for Oracle, Sybase, Informix, DB2 etc.

Type 3 A net protocol, all Java driver that translates JDBC calls into a DBMS independent net protocol which is then translated to a DBMS protocol by a server

Type 4 A native protocol all-Java driver converts JDBC calls into the network protocol used by DBMSs directly. This allows a direct call from the client machine to the DBMS server

177

Java in the ServerDevelopment Install JDK 1.3 on development machine(s)

Install any O/S patches required by vendor

We recommend that you recompile your Java UDR’s using JDK 1.3

For more information on development, please see the user guides and the Informix Developers Network website, “Java” and “DataBlade Developers” corners

178

Java in the ServerDebugging

Tip #1If you see this:

9431: Can't find system class or method or library (/vobs/tristarm/sqldist/extend/krakatoa/jre/lib/sparc/server/libjvm.so).

It may mean that you have an invalid entry for JVPJAVAM in your ONCONFIG file

179

Java in the ServerDebugging

Tip # 2If you see this in MSG_PATH file:10:33:12 Cannot load lib

/vobs/tristarm/sqldist/extend/krakatoa/jre/lib/sparc/server/libjvm.so: error : ld.so.1: oninit: fatal: libCrun.so.1: open failed: No such file or directory

This could also be a sign of an invalid entry for JVPJAVAM in your $ONCONFIG file

180

Java in the ServerDebuggingTip # 3If you see this:9431: Can't find system class or method or library

(/vobs/tristarm/sqldist/extend/krakatoa/jre/lib/sparc/server/libmath.so).

Or this in MSG_PATH: 10:33:12 Cannot load lib

/vobs/tristarm/sqldist/extend/krakatoa/jre/lib/sparc/libmath.so: error : ld.so.1: oninit: fatal: /vobs/tristarm/sqldist/extend/krakatoa/jre/lib/sparc/libmath.so : open failed: No such file or directory

You have the math library specified in your JVPJAVAM, remove it.

181

Questions???

Carlton Doe and his session – aka the warm-up band for Mark Scranton

Documents

ibm ids

differencebetween ids

release of ids

ids customers

majority of ids sales

new releases of ids

informix press books

sql compatibilityfocus