TEMENOS T24 ARCHIVING
User Guide
No part of this document may be reproduced or transmitted in any form or by any means, electronic or mechanical, for any purpose, without the express written permission of TEMENOS Holdings NV. Copyright 2005 TEMENOS Holdings NV. All rights reserved.
ARCHIVING
Table of Contents Introduction.............................................................................................................................................. 3 Application Overview ........................................................................................................................... 3 File Layout ........................................................................................................................................... 3 The Archiving Process ......................................................................................................................... 4 Timing of the Archiving process ....................................................................................................... 4 Compiling/Cataloguing programs..................................................................................................... 4 Backups............................................................................................................................................ 4 Running Archiving................................................................................................................................ 5 Introduction of New Multi Thread Archiving Processing ...................................................................... 7 Reports ................................................................................................................................................ 8 Accessing the archived information ..................................................................................................... 8 Creating enquiries to view archived information .............................................................................. 8 Viewing archived records ................................................................................................................. 9 Files.................................................................................................................................................... 10 CATEG.ENTRY - Category entries ................................................................................................ 10 DELIVERY Delivery..................................................................................................................... 10 FOREX Foreign Exchange.......................................................................................................... 11 FUNDS.TRANSFER Funds Transfer .......................................................................................... 11 MM.MONEY.MARKET Money Market ........................................................................................ 11 SPECIAL.ENTRY - Special entry files ........................................................................................... 12 STATEMENT - Statements ............................................................................................................ 12 SEC.TRADE - Security Transactions............................................................................................. 13 TELLER.......................................................................................................................................... 13
TEMENOS T24 User Guide Page 2 of 13
ARCHIVING
IntroductionApplication OverviewT24 Archiving has been designed to reclaim space by moving historic data that is no longer required. Archiving examines files for records to be archived, writes the selected records (and any associated records) to an archive file (which should reside on a separate disk, or possibly a separate machine if accessible via a network), and then deletes the records. It is also possible, if so required, to simply delete the data without archiving it.
File LayoutFor each of the sets of files for which archiving is to be run, there is a record present on the ARCHIVE file. There are three distinct sections of the record. The first contains the main data of the record and is used to specify such things as the Purge Date, Retention Period, whether data is to be removed or archived, and if so where the archived data is to be stored. The second fields (ARC.FILENAME - MODULUS) are related multi-value fields, which describe the archive files, which will be created The third section (COMPANY.RUN.IN - TIME.ENDED) are related multi-value fields, which are system populated when archiving is run. They maintain a history each time archiving is run.
Figure 1 Archive Record TEMENOS T24 User Guide Page 3 of 13
ARCHIVING
The Archiving ProcessTiming of the Archiving processIt is recommended that the archiving process be run after the Close of Business has been completed, but before users are allowed to sign on. However, since archiving removes data that is no longer used, it can be run whilst the system is being used.
Compiling/Cataloguing programsIf it is thought that the data which is going to be archived may be needed in the future for reporting purposes, then before starting archiving, ensure that the database has programs catalogued locally. The database can then be restored at a later date and reports run using the data. To check that the programs are catalogued locally, look at the CATALOG field of the SYSTEM record on the SPF. This field will be GLOBAL or null. If it is null, the cataloguing option will be LOCAL. However, care should be taken if relying on this field since, if the option on the SPF was changed from GLOBAL to LOCAL for example, only programs catalogued since the change was made will be catalogued locally. If programs are not catalogued locally, then, if the database is restored from tape the programs will need to be catalogued afterwards. The system on which archiving is to be run should be fully backed up to tape. It is important to note that archiving is a one-way process: data cannot be restored afterwards, except by fully restoring the database back to the point of pre-archiving.
BackupsBack up the area as follows:
From UNIX, change to the directory that is being backed up. For example, if the area being backed up is bnk and the directory structure is /usr/T24/bnk/bnk.run then type
$cd /usr/T24 $find bnk -print -depth | cpio -ovcB > /dev/zzz
Where zzz is the name of your backup device.
From UNIX verify the tape as follows:
$cpio -iBct < /dev/zzz
Where zzz is the name of your tape device.
TEMENOS T24 User Guide Page 4 of 13
ARCHIVING
Running ArchivingArchiving is run on one set of files at a time. There is a separate record for each set of files to be archived, the ids of which can be found in section Files. Before running archiving, it should be decided whether data should be archived and, if so, where the archived files should reside. Enter this information onto the ARCHIVE record for the set of files to be archived. If data is to be archived, files will be created with a $ARC suffix in the archive directory. For example, if special entry files are to be archived, files will be created as RE.CONSOL.SPEC.ENTRY$ARC and RE.CONSOL.SPEC.ENT.KEY$ARC. The layout of the $ARC files will be the same as the live files, with concat files being recreated as necessary. By default, the $ARC files will be created with the same type and modulus as the corresponding live files. However, if, for example, only half the data is expected to be archived, it may be desirable to create the $ARC file with a smaller modulus than the live file. This information should be entered on the ARCHIVE record. N.B. The file sizes below are examples only and should not be used.
Figure 2 Archive Record
TEMENOS T24 User Guide Page 5 of 13
ARCHIVING
NOTE: Although the file will exist in the area specified in $ARC.PATHNAME, the file is temporarily created in the run account, before being moved to the ARC directory. Therefore, there must be enough space in the run account to create the largest of the files (each file is created, moved and deleted before the next file is created). The retention period should then be determined. This can be done either by entering a specific purge date (which must be the first of a month) or by using the retention period. If todays date is 13/06/95 and a retention period of 3 months is specified, three months is calculated from the beginning of the month. Therefore, any records from before 1/03/95 will be archived. The CHECK.NO.OF.RECS field can be amended if required. This field determines the number of records to be checked before checking whether the stop indicator has been set. It also determines how often messages will be displayed showing the status of archiving (number of records processed, number of records deleted). As the size of the files to be archived differs and the processing times for different archiving jobs vary, this field will be different on each ARCHIVE record. Once the data is correct, either verify it or press F5/commit if already in verify mode. An override message will be displayed showing the date, which will be used for archiving data. At this point archiving can be terminated without any updates taking place. To continue, type Y and and archiving will start. Messages will be displayed showing the status of the archiving process, including the number of records processed and deleted. Messages are displayed depending on the CHECK.NO.RECS field on the ARCHIVE record. Information regarding which files the number of records processed and deleted are displayed for is given in section Files. To halt the process, simply input Y into the STOP.INDICATOR field on the relevant ARCHIVE record. This needs to be performed on a separate T24 login, as the login session running archiving cannot be used. Once the data has been committed the archiving process will come to a controlled halt after the next block of records has been processed (this block size is controlled by the field CHECK.NO.RECORDS).
TEMENOS T24 User Guide Page 6 of 13
ARCHIVING
Introduction of New Multi Thread Archiving ProcessingThe existing functionality is to be broadly retained but further improved by the addition of a multithreading process as a T24 service. ARC.GENERIC routine has been configured to work with this new functionality. This routine being exclusively associated to the TELLER and FUNDS.TRANSFER applications. The method by which the user activates the multi threading archiving process is modified by the addition of a new application ARC.GENERIC.REQUEST. The user is required to input the required ARCHIVE record and then verify the SYSTEM record in ARC.GENERIC.REQUEST. This will in turn activate the TSA.SERVICE agent ARC.GENERIC where the relevant records held in the history file will be deleted and written to an archive file. For FOREX and MM.MONEY.MARKET, the respective TSA.SERVICE agents ARC.FOREX and ARC.MM.MONEY.MARKET get activated. The accounting entry files are archived using the individual services CATEG.ENTRY, STATEMENT and SPECIAL.ENTRY. The method by which the user controls the number of records they wish to archive has been modified by the addition of a new field, SEL.REC.TO.PROCESS in the ARCHIVE application. Post Archiving Actions Once the archiving process is complete the file sizes of all of the files involved (both original and $ARC where applicable) should be reviewed to take into account the new number of records in the relevant files.
TEMENOS T24 User Guide Page 7 of 13
ARCHIVING
ReportsA COMO will be produced called ARC.xxxxx where xxxxx is the name of the ARCHIVE record, e.g. ARC.SPECIAL.ENTRY.
Accessing the archived informationThe archived information will be written to files with the suffix $ARC. The $ARC files use the same layout as the files from which they have been created e.g. the layout of RE.CONSOL.SPEC.ENTRY$ARC is the same as RE.CONSOL.SPEC.ENTRY.
Creating enquiries to view archived informationThe archived files can be viewed using the T24 ENQUIRY utility, by creating an enquiry to view the archived file. This enquiry can be a copy of an enquiry on the original file that was archived, setting the FILE.NAME to the name of the archived file. For example, an enquiry to view the TELLER$ARC file could be based on the TELLER enquiry, changing the FILE.NAME to TELLER$ARC.
Figure 3 Creating an Enquiry to view archived information
TEMENOS T24 User Guide Page 8 of 13
ARCHIVING
Viewing archived recordsTo enable you to drilldown to view the archived records, the following fields should be added to the archive enquiry:
Figure 4 Viewing archived records
The variable ARC.FILE.NAME should be assigned to the name of the file, e.g. for the teller archive enquiry, ARC.FILE.NAME should be assigned to TELLER (enter TELLER in the OPERATION field) or FOREX as in the example above.. DEFAULT.ARCHIVE.VIEW is a generic enquiry that is supplied in T24, to enable viewing (drilldown) of the archived records.
TEMENOS T24 User Guide Page 9 of 13
ARCHIVING
FilesCATEG.ENTRY - Category entriesCategory entries are processed first, followed by the consol profit file. For category entries, CATEG.MONTH is the driving force; for consol profit, RE.CONSOL.PROFIT is the driving force. As the category entries are used for rebuilding the P&L, entries for the current financial year cannot be archived. Therefore, the purge date is validated to ensure that it is before the last financial year-end. The CATEG.MONTH file is selected in id order, i.e. by category, by date. If the date in the key is older than the purge date, all the category entries listed on the record will be removed from the category entry file, CATEG.ENTRY and the record removed from CATEG.MONTH. If there are more than 198 category entries for one category in a month, subsequent categ month records will have been created. Archiving will process all categ month records for one category for one month before checking the stop flag. Therefore, if there are three categ month records for one category for one month, all three records will be archived before the stop flag is checked. Once the categ month file has been processed, the RE.CONSOL.PROFIT file is processed, providing archiving has not been stopped. For each record on the consol profit file, the date in the id is checked against the purge date. If the date is before the purge date, the record is archived. The CHECK.NO.RECS and records processed refer to both the CATEG.MONTH file and the RE.CONSOL.PROFIT file. The records deleted refer to both the CATEG.ENTRY file and the RE.CONSOL.PROFIT file. Therefore the records processed will be a total of CATEG.MONTH records and RE.CONSOL.PROFIT records and the records deleted will be a total of CATEG.ENTRY records and RE.CONSOL.PROFIT records. As for each record on CATEG.MONTH there could be up to 200 records to archive from CATEG.ENTRY, the CHECK.NO.RECS should be set fairly low. A figure of 100 is suggested which can be changed on-line if necessary. These files are compatible with Multi-threaded processing The ARCHIVE record for category entries is CATEG.ENTRY.
DELIVERY DeliveryDelivery message archiving now utilises a multi-threaded structure. Having created the two relevant ARCHIVE records, DELIVERY.IN and DELIVERY.OUT the user activates the archiving process via ARC.GENERIC.REQUEST then the TSA.SERVICE agents ARC.DELIVERY.IN and ARC.DELIVERY.OUT process the request. The relevant records held in the DE.I.HEADER.ARCH and DE.O.HEADER.ARCH history files are the driving force as to which records are to be archived. For each record that is to be removed from the DE.I.HEADER.ARCH and DE.O.HEADER.ARCH files, associated records are removed from the DE.I.HISTORY and DE.O.HISTORY files and the key is removed from the concat files, DE.I.HISTORY.QUEUE and DE.O.HISTORY. QUEUE.
TEMENOS T24 User Guide Page 10 of 13
ARCHIVING
NOTE: The archiving of delivery only works on completed messages on the delivery archive files, i.e. records that have been copied to delivery archive files and removed from the live delivery files. This is done in the delivery end of period programs, DE.MM.O.END.OF.PERIOD and DE.MM.I.END.OF.PERIOD. Therefore, the archiving of delivery will only remove records provided the delivery end of period has been run. Records archived from the delivery header archive files will be written to DE.O.HEADER$ARC and DE.I.HEADER$ARC. These files are compatible with Multi-threaded processing The ARCHIVE records for delivery are DELIVERY.IN and DELIVERY.OUT.
FOREX Foreign ExchangeThe FOREX history file is processed to produce a list of contract ids. For each contract, if there is no corresponding record on the live FOREX file and the date from the DATE.TIME of the last history record is before the PURGE.DATE/RETENTION.PERIOD from the ARCHIVE record; all history records for the contract are archived. These files are compatible with Multi-threaded processing The ARCHIVE record for the foreign exchange history file is FOREX.
FUNDS.TRANSFER Funds TransferThe FUNDS.TRANSFER history file is processed to produce a list of contract ids. For each contract, if there is no corresponding record on the live FUNDS.TRANSFER file and the date from the DATE.TIME of the last history record is before the PURGE.DATE/RETENTION.PERIOD from the ARCHIVE record, all history records for the contract are archived. These files are compatible with Multi-threaded processing The ARCHIVE record for the funds transfer history file is FUNDS.TRANSFER.
MM.MONEY.MARKET Money MarketThe MM.MONEY.MARKET history file is processed to produce a list of contract ids. For each contract, if there is no corresponding record on the live MM.MONEY.MARKET file and the date from the MATURITY.DATE of the last history record is before the PURGE.DATE/RETENTION.PERIOD from the ARCHIVE record, all history records for the contract are archived. Money Market contracts can have many other associated files, which require tidying up to avoid data corruption. Therefore the following files: MM.PAYMENT.ENTRY$HIS, MM.RECEIPT.ENTRY$HIS, LMM.ACCOUNT.BALANCES.HIST, LMM.HISTORY.LIST and LMM.SCHEDULES.PAST.HIST which could contain references to the Money Market contract on the history file are checked and archived as necessary. These files have been made compatible with Multi-threaded processing The ARCHIVE record for the money market history file is MM.MONEY.MARKET.
TEMENOS T24 User Guide Page 11 of 13
ARCHIVING
SPECIAL.ENTRY - Special entry files RE.CONSOL.SPEC.ENTRY is the driving file. When archiving is run on special entry files, records older than the specified date will be removed from RE.CONSOL.SPEC.ENTRY and the reference tothe record will be removed from the concat file, RE.CONSOL.SPEC.ENT.KEY. The CHECK.NO.RECS, records processed and records deleted all refer to the special entry file, RE.CONSOL.SPEC.ENTRY. As the processing for each record is fairly quick and simple, the CHECK.NO.RECS can be set higher for this ARCHIVE record than for others. A figure of 10,000 is suggested which can be changed on-line if necessary. NOTE: The archiving process for special entries starts with a select of the special entry file. As this file can be very large, this select could take some considerable length of time. These files and are compatible with Multi-threaded processing The ARCHIVE record for special entries is SPECIAL.ENTRY.
STATEMENT - StatementsFrequency 1 statements are processed first, followed by frequency 2 statements. For frequency 1 statements, the ACCT.STMT.PRINT file is the driving force; for frequency 2 statements, the ACCT.STMT2.PRINT file is the driving force. For each account record on the ACCT.STMT.PRINT file, all statement ids are processed except the last one, so that one statement is always left for frequency 1 and frequency 2 statements. However, as soon as there is an outstanding entry on a statement, no more statements are processed for that account. For each statement id, the corresponding record is read from the STMT.PRINTED file. Each entry on the statement is checked to ensure that it is does not appear on the most recent frequency 2 statement (which is not deleted) and that it is not on the ACCT.STMT.ENTRY or ACCT.STMT2.ENTRY record(all entries for this account which still have to be printed). Providing the entry has been printed, the record is then read from the STMT.ENTRY file, to ensure that the booking date is before the purge date and that the value and exposure dates are not in the future. Finally, AC.STMT.HANDOFF file is checked to ensure the record is not still present on it. Providing all entries on a statement match these criteria: the entries are deleted from the STMT.ENTRY file, the statement printed record is deleted from STMT.PRINTED file, the statement details are removed from the STMT.PRINTED file, the statement record is removed from the AC.STMT.HANDOFF$HIS file and the statement details are removed from the ACCT.STMT.PRINT file. The processing is then repeated for frequency 2 statements. The CHECK.NO.RECS and records processed refer to the ACCT.STMT.PRINT and ACCT.STMT2.PRINT files. The records deleted refer to the statement entry file, STMT.ENTRY. As the processing for each record on ACCT.STMT.PRINT is very complex and therefore slow, since
TEMENOS T24 User Guide Page 12 of 13
ARCHIVING
all entries for all statements for an account will be checked, the CHECK.NO.RECS should be set relatively low. A figure of 100 is suggested which can be changed on-line if necessary. NOTE: The archiving process for statements starts with a select of the ACCT.STMT.PRINT file. This file contains the same number of records as the ACCOUNT file and therefore this selection should not take too long. After all frequency 1 statements have been processed, the ACCT.STMT2.PRINT file will be selected. These files and are compatible with Multi-threaded processing The ARCHIVE record for statements is STATEMENT.
SEC.TRADE - Security TransactionsInitially, the live SEC.TRADE file is processed, followed by the SEC.TRADE history file. The live SEC.TRADE file is selected. This can be a large file, and the selection process is therefore likely to take some time. Each record within the date selection is deleted from the file, providing that it does not exist on the unauthorised file. The deleted records are archived, if requested. The SEC.TRADE History file is then selected. These records are deleted if they have no corresponding record on the live file. The History records are NOT archived. The CHECK.NO.OF.RECS, RECS.DELETED and RECS.PROCESSED fields refer to the SEC.TRADE live file and the SEC.TRADE$HIS history file. Disregarding the history file, which is likely to have a minimal record count, the Sec Trade processing is fairly straightforward. An appropriate figure to be used in the CHECK.NO.OF.RECS field is the default of 1,000. These are legacy files and are not yet currently compatible with Multi-threaded processing The ARCHIVE record for securities is SEC.TRADE.
TELLERThe TELLER history file is processed to produce a list of contract ids. For each contract, if there is no corresponding record on the live TELLER file and the date from the MATURITY.DATE of the last history record is before the PURGE.DATE/RETENTION.PERIOD from the ARCHIVE record, all history records for the contract are archived. These files are compatible with Multi-threaded processing The ARCHIVE record for the teller history file is TELLER.
TEMENOS T24 User Guide Page 13 of 13
