IDC/CD Sender/SDD 5. October 2006 English only CD Sender Software Design Description This document defines the CD Sender software design description. The software design will be used as the basis for implementation. Summary CD Sender is a software system that can send continuous seismoacoustic data received from IMS stations (or from a data centre forwarding data from IMS stations). The software is able to send connection requests and establishes a connection with a data consumer. Multiple simultaneous connections can be established. For each connection, the software is able to send specific data channels and re-sign modified frames. A number of output files can also be generated.
This document is posted to help you gain knowledge. Please leave a comment to let me know what you think about it! Share it to your friends and learn new things together.
Transcript
IDC/CD Sender/SDD 5. October 2006
English only
CD Sender Software Design Description
This document defines the CD Sender software design description. The software design will be used as the basis for implementation.
Summary
CD Sender is a software system that can send continuous seismoacoustic data received from IMS stations (or from a data centre forwarding data from IMS stations). The software is able to send connection requests and establishes a connection with a data consumer. Multiple simultaneous connections can be established. For each connection, the software is able to send specific data channels and re-sign modified frames. A number of output files can also be generated.
IDC/CD Sender/SDD Page 2
5. October 2006
Document History
Version Date Author Description
0.1 23 January 2003 Stephen Lloyd Initial template.
0.2 4 February 2003 Johannes Schreitl
First issue.
0.3 11 February 2003 Johannes Schreitl
Changes following from comments from John Coyne and Stephen Lloyd.
0.4 19 February 2003 Johannes Schreitl
Changes following from comments from John Coyne and Stephen Lloyd. Adding of shutdown and startup scenarios, removal of history features.
0.5 05 November 2003 Stephen Lloyd Conversion to new template and other updates to bring document up to date at start of new CD Sender contract with Siemens.
0.6 01 March 2004 Gerald Klinkl Convert external graphics to inline drawings
0.7 07 January 2005 Gerald Klinkl Document update
0.8 27 January 2005 Gerald Klinkl Add John’s comments
0.9 15 November 2005 Gerald Klinkl Document update
This document applies to the CD Sender version 1.1.
1.2. System overview
CD Sender is a software system that can send continuous seismoacoustic data received from IMS stations (or from a data centre forwarding data from IMS stations). The data will be sent to data consumers according to the formats and protocols defined in IDC (2002), Formats and Protocols CD1.0 and IDC (2002), Formats and Protocols CD1.1.
The software is able to send connection requests and establishes a connection with a data consumer. Multiple simultaneous connections can be established.
For each connection, the software is able to send specific data channels and re-sign modified frames. A number of output files can also be generated.
The primary user of the software will be the International Data Centre (IDC) Division and the International Monitoring System (IMS) Division at the Comprehensive Nuclear-Test-Ban Treaty Organization (CTBTO). In addition, it is expected that the software will be used at National Data Centres (NDCs). The software is available to State Signatories.
The software will continue to develop in the future. This is required in order to follow expected advances in the formats and protocols (for example, the addition of command and control messages).
1.3. Document overview
This document defines the CD Sender software design. The software design will be used as the basis for implementation. The design is based upon the software requirements described in IDC (2003), CD Sender Software Requirements Specification.
This document is mainly intended for developers, maintainers and documentation writers. It is also of interest to project management, requirements analysts, quality assurance staff and user representatives.
The design is described in terms of a set of connected entities. An entity is an element (component) that is structurally and functionally distinct from other elements and that is separately named and referenced. Entities may be sub-systems, data stores, modules, programs, processes, or object classes. Entities may be nested or form hierarchies.
Each entity is described in terms of requirements and design decisions.
Each mandatory, testable requirement is stated using the word shall. Therefore, each shall in this document should be traceable to a documented test. Each mandatory, non-testable requirement is stated using the word will. Each recommended requirement is stated using the word should. A permissible course of action is stated using the word may. This convention is used in ISO/IEC (1995), ISO/IEC12207.
Each mandatory, design decision is stated using the word shall or will. Each design recommendation is stated using the word should. A permissible course of action is stated using the word may. This convention is used in ISO/IEC (1995), ISO/IEC12207.
IDC/CD Sender/SDD Page 6
5. October 2006
This document is compliant with IDC (2003) Software Design Description Template, IDC (2002), Software Documentation Framework and the CTBTO/PTS (2002), Editorial Manual.
IDC/CD Sender/SDD Page 7
5. October 2006
2. EXTERNAL INTERFACES
All external interfaces visible to the user (command line interface, configuration options, input files and output files) are described in IDC (2004), CD Tools Software User Guide.
2.1.Libraries
2.1.1. Public Libraries
Table 1. Public Libraries
Type Description
Standard C Required for all C programs and used automatically when a C program is linked.
POSIX threads Required for the POSIX threads of CD Receiver (-lpthread).
Socket Used for the network communication of CD Receiver (-lsocket and -lnsl; included in the standard C library on Linux).
Mathematic Used for simple arithmetic by CD Receiver (-lm; most of the functions are included in the standard C library on Linux).
Crypto Used for authentication verification. CD Receiver uses a library coming with the openssl package (-lcrypto).
LDAP Used to retrieve certificates from an LDAP directory server. CD Receiver uses a library coming with the openldap package (-lopenldap).
2.1.2. IDC Libraries
Table 2. IDC Libraries
Type Description
cancomp Used for Canadian compression/uncompression and is part of the CD-Tools distribution.
paridc Used for reading and processing parameter files and is part of the CD-Tools distribution.
idcsyslog Used logging to syslog and for debug messages and is part of the CD-Tools distribution.
cd Used for decoding of CD frames, signing and verifying frame data and is part of the CD-Tools distribution
2.2. Rationale
To take advantage of concurrency within the OS and the hardware the CD Sender has been designed as a multithreaded implementation. The multithreading approach guarantees full use of multiprocessor systems and exploits the high level of inherent parallelism.
2.3. General Implementation
The sender has been written in ANSI C and has been designed to run on UNIX and Linux systems.
IDC/CD Sender/SDD Page 8
5. October 2006
2.3.1. Requirements
The requirements for the CD Sender are defined in IDC 2004, CD Sender Software Requirement Specification.
2.3.2. Design decisions
The CD Sender is designed as a multithreaded application with a main thread, a scheduler thread, an optional command thread, an optional certificate thread, and one or more handling threads. The main thread is responsible for initialisation and handling of signals, the scheduler thread checks for new data and informs the handling threads about new data, and the handling threads manage the connections to the data consumers. Threads are synchronized using mutex locks. The application is designed to avoid using locks where possible using a design which does not require locking. Another approach to avoid mutex locks used in the CD Sender is the use of ‘atomic operations’. An ‘atomic operation’ is an operation which can be executed as a single CPU statement and needs therefore no mutex lock. Changing a pointer value is an example for an ‘atomic operation’. If the critical value cannot be changed by an ‘atomic operation’ (e. g. modifying a string value), the critical value is transformed to an atomic type (e. g. a pointer alternating pointing to two strings). An example where this transformation could be used is a file path. The file path is a string used by all handling threads. It cannot be change directly, because one or more threads may use the string while it is being modified. This would lead to unexpected results. But if all threads use a pointer to a string and the new value is copied into another string and then the pointer value is changed, it is to use for all threads. Another method to avoid mutex locks in the CD Receiver is to design the code to have only one writer (thread) for a particular state of an object. An example for this method is the per station certificate cache. It is initialized by the main thread (during pre-load, no other threads started) or by the scheduler thread (when the production line is initialised) and modified by the certificate thread (only if the cache has the proper state). Functions of common interest are designed as library functions and are implemented in a separate library, libCD. This library is used also from CD Receiver and CD2W. The CD Sender will require a Concise List of Frames (clf) file for each binary data file to send and an index file which contains information where the Concise List of Frames files are located to be able to send CD frames. This decision was made to avoid re-parsing of binary files, because the sender has to know size, offset and nominal time of frames in the binary files to sort the frames according to the current sending mode. No protocol transformation from CD-1.0 format into CD1.1 format or vice versa will be supported.
Deleted: Receiver
Deleted: listener
Deleted: during first
Deleted: connection request
IDC/CD Sender/SDD Page 9
5. October 2006
2.3.3. External Interfaces
The CD sender uses the following external interfaces:
Optional
File system
Configuration
Data Station
wfdisc records
Certificates
write
concise listof frames
Syslog
text represen-tation offrames
received
Leased Lines,Internet, etc.
CD1.0CD 1.1
SystemMessages
CD Receiver
Required
CD Frames
Index File
write
Main
CD Sender
Monitor
read
Configuration
read
write read/write
Syslog
Data Consumer
Leased Lines,Internet, etc.
CD 1.0CD 1.1
Receiver - SenderOverview
Internal and External Interfaces
write read
Sch
edul
e
??
Wor
ker
Internalreference
list
write
concise list offrames received
write
Productionline list
DataConsumer
File
sentCD frames
text represen-tation of
sent/receivedframes
Figure 1: CD Receiver – CD Sender Overview Internal and External Interface
IDC/CD Sender/SDD Page 10
5. October 2006
3. PROCESSING ENTITIES
3.1. Main processing entity
The main thread of the sender is primarily concerned with system configuration, starting the scheduler, and the optional command and certificate threads.
3.1.1. Dependencies
Figure 2: Thread Overview Thread Model
3.1.2. Overview
3.1.3. Design Decisions
3.1.3.1. Process flow Main Thread
Worker Thread 1
Scheduler
Worker Thread 2
Main
Certificate
Worker Thread 3
Command
IDC/CD Sender/SDD Page 11
5. October 2006
Figure 3: Process Flow Main Thread
3.1.3.2. States of Main Thread
shut- down
Configure Sender
Start Command Thread
Start Certificate Thread
Start Schedule Thread
Wait for user termination
Wait until all worker threads are stopped
Shutdown all threads
Start
CD- Sender Main Thread Process Flow
IDC/CD Sender/SDD Page 12
5. October 2006
Figure 4: States of Main Thread
3.2.Schedule processing entity
3.2.1. Overview
The scheduler thread of the sender is spawned by the main thread and can run in two modes: fixed interval mode or continuous mode. The scheduler thread spawns the worker threads. The number of spawned worker threads depends on defined entries in the Production Line File.
3.2.2. Dependencies
Please see Figure 2 on page 5.
3.2.3. Design decisions
CD Sender will support manually configured and dynamic created production lines. A manual configured production line is a production line with a defined data provider and defined data consumer. A dynamic production line does not have a defined data provider. It only have a defined data consumer. For each manually configured production line, the scheduler thread will spawn a worker thread at startup time For each dynamic production line, the scheduler thread will spawn a worker thread for each data provider found in the index file. The number of worker threads depends on defined relations in the Production Line File. The scheduler thread will read the configured index file periodically and will pass an initial list of clf files to process to each new started worker thread. The scheduler thread will also update a worker threads clf file list if a new clf file to process is available. The clf file list is implemented as a linked list of structures and a modification of the clf file list must be protected by a mutex lock. This is caused by the fact that the list has to be modified by the worker thread when a clf entry is removed and by the scheduler thread when a clf entry is added.
Initialize
Wait for termination
request or signal to shutdown
Wait for all threads to shutdown
User termination request
All threads to shutdown or second user termination request received
Start application
IDC/CD Sender/SDD Page 13
5. October 2006
The schedule thread can run in several modes: (a) Continuous mode
After startup the scheduler thread searches for frame summary (clf) files in the CD-Receiver generated index files which are created before look back scheduler time interval defined hours backwards from the current start time, generates a list of clf files, and notifies the related worker threads and passes the appropriate clf files to each worker thread. The name of index file depends on the well known port of the related receiver, for more details please see IDC 2004, CD Tools SUG. The search path for the index file depends on the configured file path . Each sleep time scheduler defined seconds, the scheduler thread looks into the index file to see if new clf files are available. The related worker threads will be notified of any newly available clf file.
(b) Fixed Internal Mode In this mode the schedule thread searches in the index files for frame summary (clf) files created after start time and before end time, generates a list of clf files, notifies the related worker threads and passes the appropriate clf files to each worker thread. Then the scheduler thread waits until all worker threads have sent their data and are being stopped. Start time and end time are passed as command line parameters to the sender. If no end time parameter is provided, the current date and time is used as end time.
(c) Auxiliary Station Mode In this mode the schedule thread will behave like in the Continuous Mode, but will not start worker threads immediatelly. Instead it will put all productions lines into stopped state and wait for a “data request” command. The scheduler thread will start the worker thread only when a “data request” command is received. The worker thread will send the requested data, puts the production line into stopped state again and shut down after the requested data have been sent. The schedule thread will start the worker thread again when a new “data request” command arrives. As long as the requested data is found in the clf files defined by current time and lookBackTime , the same data may be requested more than one time. No restart files are written in Auxiliary Station Mode.
Formatted
IDC/CD Sender/SDD Page 14
5. October 2006
3.2.3.1. Process Flow of Scheduler Thread
Figure 5: Process Flow of Scheduler Thread
Start
Shut down
sleep
Search in index files for data files to send
Configure Thread
mode
Data files found
Write file names into buffer of working
threads
Shutdown processing
Shut down
Fixed Interval mode
No
Continuous/ Auxiliary station mode
Spawn worker Threads
Write file names into buffer of
working threads
Data available
Shut down processing
Look in current index file for data
files to send
Write file names into buffer of
working threads
Shut down
no
Termination request
For each relation data provider vs. Data consumer one worker thread exist; Not in auxiliary station mode
Index is written by CD- Receiver
Wait for worker threads finished
Search in index files for data files to send
For each relation data provider vs. data consumer one worker thread exist
Spawn worker Thread(s) if necessary
In auxiliary station mode is the station into
STOPPED state when no
data request is pending
IDC/CD Sender/SDD Page 15
5. October 2006
3.2.3.2. States of Scheduler Thread
Figure 6: States of Scheduler Thread
Initialize
Processing shutdown
Search for data files
Start worker threads
Inform worker thread
Wait for new data
Wait for worker threads finished
Start application
Continuous mode
Termination request
New .clf file available
Check new .clf file available
Data files found
Processing shutdown
Termination request or all data are sent
Fixed interval mode
IDC/CD Sender/SDD Page 16
5. October 2006
3.2.3.3. Process Flow of Scheduler Thread Start Up in Continuous Mode
Figure 7: Scheduler Thread Start Up in Continuous Mode
Spawned and initialized by main thread
Search in index files updated by CD
Receiver for .clf files using look back
parameter
spawn worker thread
Read production line file
For each production line
Create a station object
Create a reference list of found clf files
End for each production line
sleep
Check index file for new clf files
For each production line
Shutdown
Update reference list of clf files
End for each production line
wait for all worker threads to shut down
No
Yes
spawn worker thread if necessary
IDC/CD Sender/SDD Page 17
5. October 2006
3.2.3.4. Process Flow of Scheduler Thread Start Up in Auxiliary Station Mode
Figure 8: Scheduler Thread Start Up in Auxiliary Station Mode
3.2.3.5. Process Flow of Scheduler Thread Start Up Fixed Interval Mode
Spawned and initialized by main thread
Search in index files updated by CD
Receiver for .clf files using look back
parameter
Read production line file
For each production line
Create a station object
Create a reference list of found clf files
End for each production line
sleep
Check index file for new clf files
For each production line
Shutdown
Update reference list of clf files
wait for all worker threads to shut down
No
Yes
spawn worker thread Put station in stopped
state
Data request pending
End for each production line
No
Formatted: Bullets andNumbering
Formatted: Bullets and
Numbering
IDC/CD Sender/SDD Page 18
5. October 2006
Figure 9: Scheduler Thread Start Up Fixed Interval Mode
A clf file list is never shared between different worker threads, regardless if worker threads use the same data provider. This approach simplifies the design, because it requires only a 1:1 lock of the clf file list. On the other hand there is some performance cost, because each clf file must be parsed in each worker thread.
Spawned and initialized by main thread
Search in index files updated by CD
Receiver for .clf files using look back
parameter
spawn worker thread
Read production line file
For each production line
Create a station object
Create a list of found clf files
End for each production line
wait for all worker threads to shut down
Deleted: reference
IDC/CD Sender/SDD Page 19
5. October 2006
Figure 10: Data flow scheduler to worker threads
scheduler
Worker Worker
Index file
Internal reference list
clf clf
clf file list
Internal reference list
clf file list
Formatted: Bullets andNumbering
IDC/CD Sender/SDD Page 20
5. October 2006
3.2.3.6. Process Flow of Scheduler Thread Shut Down
If a shutdown notification is reached by the scheduler thread, the scheduler thread will wait until all worker threads have been terminated.
Figure 11: Scheduler Thread Shut Down
Write log
exit
Notification for shutdown
received
wait for all worker threads to shut down
IDC/CD Sender/SDD Page 21
5. October 2006
3.3. Worker processing entity
The worker thread is spawned by the scheduler thread. For each Production Line a worker thread is spawned. Each worker thread manages a single connection to a data consumer. In continuous mode a worker thread remains active until the sender is shut down and is notified by the scheduler thread if new clf files are available for processing.
3.3.1. Overview
In the worker thread there are several functions implemented.
(a) Parse the CD Receiver generated clf files for data frames to send and store them in an internal reference list.
(b) Filter data sub channels by defined rules and assemble, if necessary, new frames containing only sub channels which match the filter condition.
(c) Maintain the connection to a data consumer. (d) Send CD data to data consumer. (e) Logging of sent and received frames. (f) Compute the correct frame to send after a re-start
3.3.1.1. Internal Reference List
The internal reference list is a linked list of frame descriptions stored in memory. The list is either sorted by nominal time or unsorted depending on processing mode. Information related to the current state of the internal reference list will be stored to a restart file every reference list store interval seconds. The name of the restart file reflects the relation between data provider and data consumer: .<data_provider_name>_<data_consumer_name>.irl Example:
.WRA_BRNDC.irl
The restart file will contain information about the current processing mode (FIFO, LIFO or as arrived), the last clf file parsed, the last frame sent, and a list of unacknowledged CD-1.1 frames. Please see IDC 2004, CD Tools SUG, for more details concerning the format of the restart file. This list is used for keeping track of data to be sent to the data consumer. The scheduler thread provides a list of clf files for the worker thread at startup time. The worker thread parses the clf files to find Data and Data Format frames to send and uses information from the restart file to computes the correct restart point (the first frame to send) and adds unsent or not acknowledged frames to the internal reference list. The last step is to sort the internal reference list according the current processing mode. Entries in the internal reference list are removed when the restart file is updated or when the internal reference list is compacted. Only entries which have state acknowledged are removed from the internal reference list. Changing the state to acknowledged is done when:
Formatted
Formatted
Deleted: The worker computes the correct restart point (the first unset frame) using information from the restart file Then the
Deleted: those
IDC/CD Sender/SDD Page 22
5. October 2006
o the state of the frame is sent, but not acknowledged and the internal reference list is compacted (only CD-1.0 connections)
o an AckNack Frame is received which acknowledges the frame (only CD-1.1, not implemented yet)
o the maximum retry count is reached
Sent, not acknow- ledged
Not sent
Not sent, Retry count > 0
Not sent, requested
Acknow-ledged
Frame sent
Frame sent
Data Request Command
Frame sent
Reconnect/ Restart
Max. retry count reached, IRL compacted or frame acknowledged
Figure 12: Frame states
The internal reference list is periodically updated in continuous mode.
3.3.2. Restart
To prevent loss of data when the connection is closed, the worker thread will write all unacknowledged frames to the restart file. For CD-1.0 connections, the last N frames will always be treaded as unacknowledged. For CD-1.1 connections, frames are acknowledged by AckNack frames sent by the peer CD receiver (not implemented yet). For N a value of 10 seems to be reasonable (to change it search for NBR_RESENT_FRAMES in the source code). This behavior can cause sending previous sent frames when the sender performs a restart. The worker thread uses the information in the restart file to determine which frames have to be sent when the sender restarts. If no restart file is available the sender will start to send data received since the begin of the current day. If the restart information is too old, all frames found in the clf file list are sent. No restart information is written when a “data request” command is being processed.
Formatted: Bullets and
Numbering
Formatted: Bullets andNumbering
Deleted: the complete frame has been sent (CD-1.0) or when the sent frame has been acknowledged (CD-1.1).
Deleted: go
Deleted: back in the internal reference list before writing or updating the information in the restart file
IDC/CD Sender/SDD Page 23
5. October 2006
3.3.2.1.1. Differences between continuous mode, fixed interval mode and auxiliary mode
Differences between continuous mode, fixed interval mode and auxiliary mode are:
• there is no update of the clf file list (and therefore no update of the internal reference list) in fixed interval mode.
• the worker thread will shutdown in fixed interval mode if the internal reference list becomes empty, i.e. after all specified data have been sent.
• the worker thread will shutdown in auxiliary mode if all requested data have been sent, but will started again if a new data request command is received.
3.3.2.1.1.1. FIFO processing mode
In this mode the reference list is sorted in ascending frame nominal time order. The oldest frames will be sent first.
3.3.2.1.1.2. LIFO processing mode
In this mode the reference list is sorted in descending frame nominal time order. The newest frames will be sent first.
3.3.2.1.1.3.As arrived processing mode
In this mode the reference list is not sorted. The frames will be sent in the same order as they are received.
3.3.3. Design decisions
A worker thread will not terminate if the connection has been closed or cannot established. A worker thread will terminate when a disconnect or stop command is received. Update of the clf list must be protected by a mutex lock.
3.3.3.1. Process Flows
Formatted: Bullets andNumbering
Formatted: Bullets and
Numbering
Formatted: Bullets and
Numbering
Formatted: Bullets and
Numbering
Formatted: Bullets and
Numbering
Formatted: Bullets and
Numbering
Deleted: and
Deleted: The two d
Deleted: and
IDC/CD Sender/SDD Page 24
5. October 2006
Figure 13: Process Flow of Worker Thread (overview)
Start
Block A Process restart file
have clf file list
fixed interval mode
No
Yes
No Yes
have clf file to process
have frames to send
Yes
Shut down
fixed interval mode
Yes
Shut down
Save internal reference list
No
Shutdown pending
Block D Connect if
unconnected, send frames and process
received data
Yes
(re-)parse clf file and update reference list
No
Block C Wait with timeout for
received data if connected
No
Block B Check for shutdown and wait until data to send is available if
unconnected
Shutdown pending
IDC/CD Sender/SDD Page 25
5. October 2006
The first action when a worker thread starts is to check for and process the restart file. If a restart file exists, an attempt is made to build an internal reference list containing the same unsent frames as before shutdown of the production line. If the shutdown happened before the (configured) look back time of the scheduler thread, the internal reference list will not contain frames received before the look back time.
Figure 14: Process Restart File (Block A)
After processing the restart file, the worker thread checks if a shutdown is pending. If a shutdown is pending, the worker thread terminates. Otherwise it checks if the connection to
parse clf files until restart
point reached
Read restart information
Restart file valid
Search restart point in clf list
Restart point found
Log warning
Processing mode
skip clf files until restart
point reached
parse restart clf file
parse remaining clf
files
Yes
Yes
Yes
No
No
No
FIFO, LIFO As arrived
Restart file exists
Block A
Deleted: To avoid performance problems when saving huge internal reference list, especially during restarts, the restart file will not contain the same information as stored in the internal reference list. Instead it contains information of the last processing mode, the last parsed clf file, the last sent frame and a list of unacknowledged frames (only CD-1.1)
IDC/CD Sender/SDD Page 26
5. October 2006
the data consumer is established, if a clf file is queued to be processed or if the current clf file has grown since the last parsing. If there is nothing to do, the worker thread waits sleep time worker thread before it checks for new data.
Have data to send
connected
Shutdown pending
No
Yes
No
wait
No
Yes
Yes
Figure 15: Wait until data to send is available (Block B)
If data to send is available, all clf files found in the clf file list will be parsed. If only the current clf file has grown, all the new lines will be parsed. Frames to send found in the clf file will be added to the internal reference list. New elements will always be inserted as first element. If the internal reference list does not contain unsent frames after parsing of the clf file and the connection is established, the worker thread waits sleep time work for received data. If data are received, the Sender tries to read a complete frame and continues with Block B.
Block B
Deleted: until new data arrives.
Deleted: the current
Deleted: (re-)
IDC/CD Sender/SDD Page 27
5. October 2006
Figure 16: Wait with timeout for received data (Block C)
If the internal reference list contains unsent frames after parsing of the clf file, an attempt is made to establish a connection, if necessary, and frames, marked as unsent are sent. After each frame sent the worker thread checks for and processes received data. If the connection cannot be established, the worker thread waits sleep time retry, checks for a pending shutdown and tries again to establish the connection.
connected
Wait for incoming data
or timeout
Data received
Process frame
Yes
Yes
No
No
Block C
IDC/CD Sender/SDD Page 28
5. October 2006
Figure 17: Connect, send data and checks for received data (Block D)
If the connection is established and data to send are available, CD Sender sorts the internal reference list, if necessary, and starts sending frames beginning with the first element in the internal reference list (Block E). Before a frame is sent, CD Sender checks for incoming data and reads a single frame when incoming data are pending. Then the first element on the internal reference list is sent. On CD-1.0 connections, a check is made if it is necessary to send a Data Format Flush Alert Frame and a Data Format Frame before a Data Frame is sent. The sent frame will be marked as sent (CD-1.0) or sent, not acknowledged (CD-1.0) and the sender checks if reference list store interval is reached. In that case it will compact the internal reference list and update the restart file. Then the sender checks for new data arrived in the current clf file (or in a new clf file if a file switch has occurred in the meantime). If no user shutdown request is pending, the next check for incoming data is done and the next frame
connected try to
connect
retryWait
Block E Send frame and
check for and process received
data
yes yes
Irl empty
connected
yes
Pending shutdown
no
no
no
Block D
IDC/CD Sender/SDD Page 29
5. October 2006
is sent. This is repeated until no more frames to send are on the internal reference list, or a user shutdown request is pending or the connection is closed by the remote site.
Sort IRL if necessary
Have more frames to send
Send DFF
Check for and process
incoming data
Send DF
Compress IRL and update
restart file
yes
no
Must send DFF
N data frames sent
no
yes
no
yes
yes
no
Connection closed, shutdown request
Shutdown pending or connection closed
Check for new arrived
data and update IRL
IDC/CD Sender/SDD Page 30
5. October 2006
Figure 18: Send frame and check for and process received data (Block E)
3.3.3.2. States of Worker Thread
Figure 19: States of Worker Thread
3.3.3.3. Process Flow of Worker Thread Shut Down
When the worker thread gets a notification that a shutdown request is pending, it
• sends an Alert frame and closes the connection when it is connected; • updates the restart file when data have been sent; • clear the internal reference and exits
The parsing of a clf file is not stopped when a user termination request is pending. The pending user termination request is served when the parsing of the clf file is completed.
Initialize Wait for new data
Start application
user Termination
All data sent in fixed internal mode, requested data sent in auxiliary mode or shutdown request
Check clf file
Send frame
Processing received frames
Processing shutdown
Data sent in continuous mode
Update reference list
Determine data available to send
Frame received
Formatted: Bullets and
Numbering
Formatted: Bullets and
Numbering
Deleted: and the clf file list
IDC/CD Sender/SDD Page 31
5. October 2006
Figure 20: Worker thread shutdown processing
3.4. Certificate processing entity
The certificate processing entity is the same as used for CD Receiver and is described in IDC 2004, CD Receiver Software Design Description.
connected
Update restart file
Shutdown notification
Clear IRL and clf list
Shut down
yes
Send alert frame and disconnect
no
IDC/CD Sender/SDD Page 32
5. October 2006
INTERFACE ENTITIES
3.5.Configuration interface entity
3.5.1. Overview
The CD Sender is using several text files to import configuration data. If any configuration file is not found, a warning will written to stderr and the software will stop.
3.5.2. General
This chapter gives only a rough overview of the interface entities. You will find a detailed description of the command line interface and the format of the configuration files in IDC 2004, CD Tools SUG. The CD Sender will use libCD functions for reading and parsing common used files like the configuration file, the station file and the key preload file. File parsing functions used by the CD Sender, which are of common interest will be designed as library functions. Examples for such functions are parsing of the index file and parsing of clf files.
3.5.2.1. Command Line Interface
The CD Sender accepts one mandatory argument, which is the name of configuration file and two optional arguments, which are start time and end time for fixed mode processing. This configuration file contains all configuration parameter and references to additional configuration files. (e.g. name of Production Line File) The parameters of the configuration file are described below. If the sender is running typing ctrl-c can stop it. If no configuration file is specified an error message will written to stderr and the software will stop. If in the configuration file some parameters are missing, a warning will written to stderr and syslog, and default values will be used.
3.5.2.2. Additional Mandatory Configuration Files
Additional mandatory configuration files are: (a) Data Consumer File (b) Production Line File If either of these files are missing or are invalid, the CD sender software will write an error message to stdout, syslog and stop. In the Data Consumer File there is at least one configuration of Data Consumer required. Also in Production Line File there is at least one configuration of a Production Line required. If these minimum requirements are not available an error message will written to stderr, syslog and the software will stop.
3.5.3. Operating mode parameters
IDC/CD Sender/SDD Page 33
5. October 2006
3.5.3.1. Design decisions
The operating parameters that may be specified in the configuration file are described in the following table:
Table 3. Operating mode parameters
Parameter Description
Sleep time scheduler For scheduler thread The number of seconds to sleep while waiting for new data (continuous mode only).
Sleep time worker For worker thread The number of seconds to sleep while waiting for new data (continuous mode only).
Processing mode Processing sequence of arrived data (FIFO,LIFO, as arrived)
Look back time scheduler Time in hours, the scheduler thread looks for data files after startup which are maximal older of the scheduler’s startup time when a restart file exists. Otherwise only data files created at the current day are sent.
3.5.4. Persistence parameters
3.5.4.1. Design decisions
The persistence parameters that may be specified in the configuration file are described in the following table:
Table 4. Persistence parameters
Parameter Description
Reference list store interval Number of sent frames after the restart file should be updated and the internal reference list compacted
Reference list store path Path to store area of internal reference list
3.5.5. Processing parameters
3.5.5.1. Design decisions
The processing parameters that may be specified in the configuration file are described in the following table:
Table 5. Processing parameters
Parameter Description
Processing mode Processing sequence of arrived data (FIFO,LIFO, as arrived)
IDC/CD Sender/SDD Page 34
5. October 2006
3.5.6. Input file parameters
3.5.6.1. Design decisions
3.5.6.1.1. Data Consumer File
The Data Consumer File contains a list of valid data consumer names, IP-addresses and well known ports for which connections will be established and processed.
3.5.6.1.2. Production Line File
The Production Line File contains a list of valid station names and specifies which data are sent to data consumers. Station data can be extracted and sent to one or multiple data consumers (semicolons are delimiters). Each consumer must be defined in ‘Data Consumer File’. For each relation Data Provider vs. Data Consumer, one line is required. The Scheduler thread will spawn a worker thread for each Data Provider/Data Consumer pair. Each line in the Production Line File has the following fields: (a) station name (b) data consumer name or wildcard ‘*’ for passive data forwarding (c) extraction of site rule (regular expression only) (d) extraction of channel (regular expression only) (e) station name of data consumer (not implemented yet)
3.5.6.2. Design decisions
The input file parameters that may be specified in the configuration file are described in the following table:
Table 6. Input file parameters
Parameter Description
Index file path The operating directory where the index files by the CD Receiver will be written.
port Well known port of the related CD-Receiver written index file. This value is part of the name of index file. A defined port is mandatory.
Station file Path and name of file containing names, IP address and station names which are allowed to connected to the command port of the CD Sender.
Data Consumer file Path and name of file containing the names, IP address and CD Receiver’s well known port.
Production Line file Path and name of file containing the description of which data has to be sent from each station to each data consumer and which pattern are defined to extract the proper set of sites and channels.