Manchester Conference 2003 SIR User Conference Manchester July 4 th 2003.

Manchester Conference 2003

SIR User ConferenceSIR User ConferenceManchesterManchesterJuly 4July 4thth 2003 2003

SIR Product DevelopmentSIR Product Development

Where are we now?Where are we now? Where are we going?Where are we going?

Current Release SIR2002Current Release SIR2002

Data Management Data Management – Multiple DatabasesMultiple Databases– Integrated Master ModulesIntegrated Master Modules– SQLServer for ODBC SQLServer for ODBC – Secondary IndexesSecondary Indexes– Long StringsLong Strings

Current Release SIR2002Current Release SIR2002

Application DevelopmentApplication Development– GUI BasedGUI Based– Screen PaintersScreen Painters– Unlimited PQL Program Size Unlimited PQL Program Size

(No ‘Table Space’)(No ‘Table Space’)

– SpreadsheetSpreadsheet– PQLFormsPQLForms– Internet EnabledInternet Enabled

Current Release SIR2002Current Release SIR2002

Single Generic Source on Windows XPSingle Generic Source on Windows XP Releases on Windows, Sun, HP, IBM, Releases on Windows, Sun, HP, IBM,

LinuxLinux SPRs fixed on demand - revisions on SPRs fixed on demand - revisions on

internet for downloadinternet for download Following Last ConferenceFollowing Last Conference

– Upgraded PQLFormsUpgraded PQLForms– Corrected Secondary Index IssuesCorrected Secondary Index Issues– Current version 20.23Current version 20.23

Where are we going?Where are we going?

Support Multiple PlatformsSupport Multiple Platforms Compatibility from earlier versionsCompatibility from earlier versions Ease of UseEase of Use Integrate as much as possibleIntegrate as much as possible

– No ‘M’ modulesNo ‘M’ modules– PQLFormsPQLForms– Incorporate SQL into PQLIncorporate SQL into PQL– Extend database features to remove need Extend database features to remove need

for tabfilesfor tabfiles

Where are we going?Where are we going?

VisualPQL is primary toolVisualPQL is primary tool– Sub-routines, sub-procedures, program in Sub-routines, sub-procedures, program in

program, extended size, pre-compile, program, extended size, pre-compile, functionsfunctions

– Other features use VisualPQL e.g. menu Other features use VisualPQL e.g. menu interface, painters, PQLFormsinterface, painters, PQLForms

Extend GUI FeaturesExtend GUI Features– Fonts (colour, size, etc) , Other ControlsFonts (colour, size, etc) , Other Controls– No text style screensNo text style screens

Old Forms, old editor, PQL debuggerOld Forms, old editor, PQL debugger

Suggested new featuresSuggested new features

SQL in VisualPQLSQL in VisualPQL Extended & Improved backup/restoreExtended & Improved backup/restore Multiple database data filesMultiple database data files PQL ServerPQL Server XML Input/OutputXML Input/Output GUI PQL Debugger??GUI PQL Debugger?? 32 character names??32 character names?? Sent questionnaireSent questionnaire

Feature UsedFeature Used

Visual PQL 86%Visual PQL 86%

Old Forms 38%Old Forms 38%

PQLForms 25%PQLForms 25%

Painter 20%Painter 20%

Master 30%Master 30%

ODBC 33%ODBC 33%

SQL 38%SQL 38%

Sec.Indexes 30%Sec.Indexes 30%

Web Interface 20%Web Interface 20%

Multiple DB 30%Multiple DB 30%

Tabfiles 33%Tabfiles 33%

Debugger 30%Debugger 30%

Exec Window 30%Exec Window 30%

Feature WantedFeature Wanted

SQL in PQL SQL in PQL 40%40% Improved Recovery Improved Recovery 60%60% Multiple Data file Multiple Data file 33%33% PQL Server PQL Server 20%20% XML XML 52%52% GUI PQL Debugger GUI PQL Debugger 33%33% 32 character names 32 character names 76%76%

Write In FeaturesWrite In Features Users Passwords ImprovedUsers Passwords Improved New TabulateNew Tabulate Multilingual labelsMultilingual labels PQL Design StudioPQL Design Studio Better ODBCBetter ODBC Better error messagesBetter error messages Anchor column in spreadsheetAnchor column in spreadsheet Timestamp data typeTimestamp data type Basic PQLFormBasic PQLForm Report PainterReport Painter Editor that knows about membersEditor that knows about members Audit Trails, More security Audit Trails, More security Extra GUI things, Colours in PQLExtra GUI things, Colours in PQL

Name for Next VersionName for Next Version

SIR2000 (Windows 2000 – year 2000)SIR2000 (Windows 2000 – year 2000) SIR2002 (SIR2000 ver 2 - year 2002)SIR2002 (SIR2000 ver 2 - year 2002) SIR2003 (SIR2000 ver 3) ?SIR2003 (SIR2000 ver 3) ? SIR2004 (release year) ?SIR2004 (release year) ? SIR/XP (Cross Platform)? SIR/XP (Cross Platform)? SIR/XS (eXtended System)?SIR/XS (eXtended System)? SIRDBXS ?SIRDBXS ?

SIR/XSSIR/XS

Feature WantedFeature Wanted– 32 character names 32 character names – Improved Recovery– XML – SQL in PQL – Multiple Data files– GUI PQL Debugger– PQL Server

32 character names32 character names

Database SchemaDatabase Schema– Database NamesDatabase Names– Record NamesRecord Names– Variable NamesVariable Names– Index NamesIndex Names– PasswordsPasswords

Tabfiles & TablesTabfiles & Tables Families & Families &

MembersMembers

VisualPQLVisualPQL– VariablesVariables– Sub-RoutinesSub-Routines– Sub-ProceduresSub-Procedures– LabelsLabels– Buffer NamesBuffer Names– FilenamesFilenames

Standard NamesStandard Names

– 1 – 32 characters1 – 32 characters– Start with alphaStart with alpha– CapitalizedCapitalized– Contain letters, numbers, four special Contain letters, numbers, four special

characters $ # @ _characters $ # @ _

Non-Standard NamesNon-Standard Names

– Enclosed in curly braces { }Enclosed in curly braces { }– From 1 to 30 characters (plus braces)From 1 to 30 characters (plus braces)– No translationNo translation– Any charactersAny characters– Stored without braces so sort Stored without braces so sort

normally normally

SIR/XSSIR/XS

Feature WantedFeature Wanted– 32 character names – Improved RecoveryImproved Recovery– XML – SQL in PQL – Multiple Data files– GUI PQL Debugger– PQL Server

Improved RecoveryImproved Recovery Current FeaturesCurrent Features

– JournalJournal– UnloadUnload– ReloadReload– RollforwardRollforward

ImprovementsImprovements– RollbackRollback– Transaction processingTransaction processing– Automatic RecoveryAutomatic Recovery– Journal as Audit TrailJournal as Audit Trail– Incremental BackupIncremental Backup

Journal UtilitiesJournal Utilities

Journal RecoverJournal Recover (New) (New)– Rolls forward from given placeRolls forward from given place

Journal RollbackJournal Rollback– Rolls back from given placeRolls back from given place

ItemizeItemize– Lists contents (headers)Lists contents (headers)

UploadUpload– Creates machine independent text fileCreates machine independent text file

DownloadDownload– Applies output from uploadApplies output from upload

Improved RecoveryImproved Recovery

Journaling RewrittenJournaling Rewritten Itemize rewrittenItemize rewritten Roll backRoll back Roll forwardRoll forward Auto recoverAuto recover

JournalJournal File HeaderFile Header

– Database NameDatabase Name– Version of SoftwareVersion of Software

Update Header Update Header – Data/SchemaData/Schema– Update Level Update Level

Started Started – Date/Time StartedDate/Time Started– Date/Time Finished Date/Time Finished

(for data updates)(for data updates)– User NameUser Name

Each update recordEach update record– Before update Before update – Before delete Before delete – After insert After insert – After update After update – Existing record Existing record

previous update previous update levellevel

– User (in multi-user User (in multi-user mode)mode)

Journal ProcessingJournal Processing

New PQL commands to process New PQL commands to process journalsjournals– Access to header dataAccess to header data– Access to record valuesAccess to record values

Audit TrailAudit Trail Other recovery strategiesOther recovery strategies

Journal CommandsJournal Commands

PROCESS JOURNALPROCESS JOURNAL

JOURNAL RECORD ISJOURNAL RECORD IS

PQL access to record variablesPQL access to record variables

END JOURNAL RECORD ISEND JOURNAL RECORD IS

END PROCESS JOURNALEND PROCESS JOURNAL

PROCESS JOURNALPROCESS JOURNAL [FROM = updlevel | START = date [,time] ][FROM = updlevel | START = date [,time] ] [THRU = updlevel | END = date [,time] ][THRU = updlevel | END = date [,time] ] [REVERSE][REVERSE] [FILENAME= fname ] (sr5 is the default)[FILENAME= fname ] (sr5 is the default) Return DataReturn Data

– [DATE = varname] [ENDDATE = varname][DATE = varname] [ENDDATE = varname]– [TIME = varname] [ENDTIME = varname][TIME = varname] [ENDTIME = varname]– [LEVEL = varname][LEVEL = varname]– [RECORD = varname][RECORD = varname]– [TYPE = varname][TYPE = varname]– [USER = varname][USER = varname]

ExampleExampleprogramprograminteger*4 jtypeinteger*4 jtypevalue labels jtype (1) 'Record written‘ (2) 'Before rewrite‘ value labels jtype (1) 'Record written‘ (2) 'Before rewrite‘ (3) 'After rewrite‘ (4) 'Deleted record‘ (3) 'After rewrite‘ (4) 'Deleted record‘ (-1) ‘Data header‘ (-4) ‘Schema header‘ (-5) ‘User header'(-1) ‘Data header‘ (-4) ‘Schema header‘ (-5) ‘User header'process journal TYPE= JTYPE RECORD = jrec level = jlevel FROM = 13process journal TYPE= JTYPE RECORD = jrec level = jlevel FROM = 13ifthen (jtype gt 0)ifthen (jtype gt 0). write [vallab(JTYPE)] 'Record Type ' jrec. write [vallab(JTYPE)] 'Record Type ' jrec.else.else. write [vallab(JTYPE)] 'Update level ' jlevel. write [vallab(JTYPE)] 'Update level ' jlevelfifijournal record employeejournal record employeewrite id name salarywrite id name salaryend journal recordend journal recordend process journalend process journalend programend program

Transaction ProcessingTransaction Processing

Start TransactionStart Transaction End Transaction (Commit)End Transaction (Commit) Abort TransactionAbort Transaction Journal BasedJournal Based Rollback on AbortRollback on Abort

Incremental UnloadIncremental Unload

Creates Journal of UpdatesCreates Journal of Updates– Add to end of UnloadAdd to end of Unload– Automatic processing of levelsAutomatic processing of levels– Automatic restore by reloadAutomatic restore by reload– Optional separate fileOptional separate file

SIR/XSSIR/XS

Feature WantedFeature Wanted– 32 character names – Improved Recovery– XML XML – SQL in PQL – Multiple Data files– GUI PQL Debugger– PQL Server

XMLXML

Still at concept stage – input welcomedStill at concept stage – input welcomed XML File structureXML File structure

– Text FileText File– HierarchicalHierarchical– Tags encloseTags enclose– Resembles HTML with own tagsResembles HTML with own tags

XML Names XML Names – Begin with character (or _ :)Begin with character (or _ :)– Case sensitiveCase sensitive– Allow letters, numbers - _ : . (No spaces) Allow letters, numbers - _ : . (No spaces)

Example XMLExample XML

<people><people> <person><person> <name>John D Jones</name><name>John D Jones</name> <salary>2150</salary><salary>2150</salary> <born-year>1956</born-year><born-year>1956</born-year></person></person> <person><person> <name>James A Arblaster</name><name>James A Arblaster</name> <salary>1500</salary><salary>1500</salary> <born-year>1961</born-year><born-year>1961</born-year> </person></person></people></people>

XML Design IssuesXML Design Issues

How to relate data to tags?How to relate data to tags?– Use record nameUse record name– Use variable nameUse variable name– Use variable labelUse variable label

How to generate XML?How to generate XML?– Utility / PQL Procedure/ WRITEUtility / PQL Procedure/ WRITE

How to read XML?How to read XML?– Utility / READ / New PQLUtility / READ / New PQL

Unicode – UTF-8 ISO-8859-1Unicode – UTF-8 ISO-8859-1 UTF-16 UTF-16

Example XML ProcExample XML Procretrievalretrieval

process caseprocess case

process rec employeeprocess rec employee

get vars name salary birthdayget vars name salary birthday

perform procsperform procs

end process recend process rec

end process caseend process case

xml tag = 'people' filename = ‘example.xml’xml tag = 'people' filename = ‘example.xml’

detail block tag = 'person'detail block tag = 'person'

. write name salary . write name salary

[datec(birthday,yyyy')] tag = 'born_year'[datec(birthday,yyyy')] tag = 'born_year'

end xmlend xml

end retrievalend retrieval

Example XML ReadExample XML Readprogramprogram

string*256 tagin datainstring*256 tagin datain

string*25 namestring*25 name

integer*4 salaryinteger*4 salary

date born_year ('YYYY')date born_year ('YYYY')

open input dsn='example.xml' XMLTAG = tagin error=endlabopen input dsn='example.xml' XMLTAG = tagin error=endlab

looploop

. read (input,error=endlab) datain(a*). read (input,error=endlab) datain(a*)

. if (tagin eq 'person') set name salary born_year (missing). if (tagin eq 'person') set name salary born_year (missing)

. if (tagin eq 'Name') compute name = datain. if (tagin eq 'Name') compute name = datain

. if (tagin eq 'Salary') compute salary = numbr(datain). if (tagin eq 'Salary') compute salary = numbr(datain)

. if (tagin eq 'born_year') compute born_year = datain. if (tagin eq 'born_year') compute born_year = datain

. if (tagin eq '/person') write name salary . if (tagin eq '/person') write name salary

poolpool

endlab:endlab:

end programend program

Example XML PROCESSExample XML PROCESS

programprogram

process xml filename='example.xml' process xml filename='example.xml'

. process xmltag 'person' . process xmltag 'person'

. gettags name salary born_year. gettags name salary born_year

. write name salary. write name salary

. end process xmltag. end process xmltag

end process xmlend process xml

end programend program

SIR/XSSIR/XS

Feature WantedFeature Wanted– 32 character names – Improved Recovery– XML

– SQL in PQLSQL in PQL – Multiple Data files– GUI PQL Debugger– PQL Server

SQL in PQLSQL in PQL

PROCESS SELECT variable_list | * | ALL PROCESS SELECT variable_list | * | ALL

FROM [ tabfile. ] table_nameFROM [ tabfile. ] table_name

[ database.] record_name[ database.] record_name

[INDEXED BY index_name][INDEXED BY index_name]

[WHERE condition][WHERE condition]

……

END PROCESS SELECTEND PROCESS SELECT

SQL in PQLSQL in PQL Processes from multiple data sourcesProcesses from multiple data sources

– JoinsJoins– Use WHERE to restrictUse WHERE to restrict– Full PQL conditions/expressionsFull PQL conditions/expressions– No subqueries in WHERENo subqueries in WHERE

Returns one set of named variables at a Returns one set of named variables at a timetime– Sequence as specified in commandSequence as specified in command– No aggregationNo aggregation– No GROUP BYNo GROUP BY– No UNIONNo UNION

SIR/XSSIR/XS

Feature WantedFeature Wanted– 32 character names – Improved Recovery– XML – SQL in PQL

– Multiple Data filesMultiple Data files– GUI PQL Debugger– PQL Server

Multiple Data FilesMultiple Data Files

Split the .sr3 data fileSplit the .sr3 data file No overhead for existing single file No overhead for existing single file

databasesdatabases Split based on serial key rangesSplit based on serial key ranges

– Case structure – case valuesCase structure – case values– Caseless – record type/key valueCaseless – record type/key value

Existing Data StructureExisting Data Structure

Top Level Index

Other Index Levels

Data Level

New Data StructureNew Data Structure

Top Level Index

Other Index Levels

Data Level

FILE INDEX

File IndexFile Index Contains key rangesContains key ranges Contains filenamesContains filenames Contains pointer to top level index for fileContains pointer to top level index for file New schema commandNew schema commandDATA FILES DATA FILES FILENAME = ‘ ‘ /FILENAME = ‘ ‘ /FILENAME = ‘ ‘ FROM keyvalue /FILENAME = ‘ ‘ FROM keyvalue /FILENAME = ‘ ‘ FROM keyvalueFILENAME = ‘ ‘ FROM keyvalue

Before any data addedBefore any data added Requires restructureRequires restructure

SIR/XSSIR/XS

Feature WantedFeature Wanted– 32 character names – Improved Recovery– XML – SQL in PQL – Multiple Data files

– GUI PQL DebuggerGUI PQL Debugger– PQL Server

GUI ImprovementsGUI Improvements Main Menu Replace ‘On-Fly’Main Menu Replace ‘On-Fly’ Child WindowsChild Windows

– GUI PQL DebuggerGUI PQL Debugger DISPLAY TIPBOX "text" DISPLAY TIPBOX "text"

– shows a box with text that disappearsshows a box with text that disappears ClipboardClipboard COPY/PASTE COPY/PASTE functionsfunctions Synchronized ListsSynchronized Lists Right Mouse click & Mouse over messages Right Mouse click & Mouse over messages Image buttons - tool bar buttons on dialogsImage buttons - tool bar buttons on dialogs

SIR/XSSIR/XS

Feature WantedFeature Wanted– 32 character names – Improved Recovery– XML – SQL in PQL – Multiple Data files– GUI PQL Debugger

– PQL ServerPQL Server

SIR ServerSIR Server

Similar in concept to SQLServerSimilar in concept to SQLServer– Started on networkStarted on network– Waits for users to logonWaits for users to logon– Process requestsProcess requests– Returns resultsReturns results

RequestsRequests– LogonLogon– Run named procedureRun named procedure– LogoffLogoff

Server ProceduresServer Procedures

Single System Procedure FileSingle System Procedure File Requests construct text outputRequests construct text output

– All output to standard output goes All output to standard output goes back to clientback to client

ProceduresProcedures– Contain SIR CommandsContain SIR Commands– Run PQLRun PQL

Client FacilitiesClient Facilities

LogonLogon Send Request (check error codes)Send Request (check error codes) Get output line at a timeGet output line at a time LogoffLogoff

Connect to SIRServerConnect to SIRServer

CONNECT conid SIRSERVER 'b1‘ CONNECT conid SIRSERVER 'b1‘

USER 'dad’ USER 'dad’

PASSWORD PASSWORD ‘zzzz’‘zzzz’

ERROR erridERROR errid

Send RequestSend Request

Request is only ever a CALLRequest is only ever a CALL– Family[password].member[password Family[password].member[password

(parameter list)(parameter list)

SEND REQUEST SEND REQUEST CONNECT conidCONNECT conidCOMMAND ‘myprocs.report1‘COMMAND ‘myprocs.report1‘RESULT linesRESULT lines

Positive result – lines returnedPositive result – lines returned Negative result – error with requestNegative result – error with request

Get ResultsGet Results

RETURN LINE mylineRETURN LINE mylineCONNECT conidCONNECT conidERROR erridERROR errid

e.g.e.g.Loop (or FOR I = 1,lines / ROF)Loop (or FOR I = 1,lines / ROF). return line myline. return line myline connect conidconnect conid error erroriderror errorid. if (errorid ne 0) exit loop. if (errorid ne 0) exit loop. write myline. write mylinepoolpool

TrainingTraining

New Version of Training available New Version of Training available for downloadfor download

Still at stage where looking to Still at stage where looking to refine approach NOT contentrefine approach NOT content

Feedback very welcomeFeedback very welcome

Other XS FeaturesOther XS Features

PQL based Integrated EditorPQL based Integrated Editor Files as MembersFiles as Members Master for Databases not SessionMaster for Databases not Session

SuggestionsSuggestions Kathy’s ListKathy’s List

– Allow ‘filename’ as synonym for ‘dsn’ in openAllow ‘filename’ as synonym for ‘dsn’ in open– Amend cross record functions to work with Amend cross record functions to work with

string values CNT CNTR MAX MAXR MIN MINRstring values CNT CNTR MAX MAXR MIN MINR– PWRITE to have last amended date and time PWRITE to have last amended date and time

written on procedure commandwritten on procedure command– PREAD to have option to overwrite based on PREAD to have option to overwrite based on

date/timedate/time– Compare two PWRITE to spot differencesCompare two PWRITE to spot differences– Facility to rename familyFacility to rename family

Manchester Conference 2003

SIR User ConferenceSIR User ConferenceManchesterManchester

20032003