OVERVIEW OF SERVER-SIDE PROCESSING Doug Clough SYNERGETICS Engineered Systems 12 January 2009 .

OVERVIEW OF SERVER-SIDE PROCESSING

Doug Clough

SYNERGETICS Engineered Systems

12 January 2009

www.ptagis.org

12 January 2009

Pacific States Marine Fisheries Commission

Server-Side Processing

• Data Basics (15 min)– PTAGIS Specification Document– Tagging & Interrogation Files– Core Database Tables

• PTTP – Client and Server Implementations (15 min)– Request processing– Request transformation

• FDVL – Field Data Validating Loader (30 min)– Validation and Loading– User Feedback: PTTP and Email– Loader performance report

• IDL – Interrogation Data Loader (30 min)– Validation and Loading– Loader performance report– Interrogation System performance reports

12 January 2009







12 January 2009


Data Basics

• PIT Tag Specification Document

• Tagging & Interrogation Files

• Core Database Tables

12 January 2009


Data Basics

• PIT Tag Specification Document• Tagging & Interrogation Files


12 January 2009


PIT Tag Specification Document

• PIT Tag Steering Committee (PTSC)

• Tagging & Interrogation Files– Tagging File pg 14– MiniMon Interrogation File pg 27– MULTIMON File pg 30

• Core Database Tables– Appendix F– Excludes ‘process-specific tables’

A subcommittee of the Fish Passage Advisory Committee of the Columbia Basin Fish and Wildlife Authority. The Committee is made up of technical representatives of the Basin’s fisheries agencies and tribes. The Committee's function is to provide guidance in the development of tagging methodologies and data standards and to serve as a technical forum for the operation of PTAGIS and the PIT tag data collection and data distribution system.

V. Glossary of Terms – pg 71

12 January 2009


Data Basics


• Tagging & Interrogation Files• Core Database Tables

12 January 2009


Tagging Files

• Multiple versions are anticipated– All versions maintained on server

– FDVL provides insert / update / delete capability

– Latest version is in database

• FILE TITLE Header Record– Auto-generated by P3

– xxxYYDDD.zzz• xxx Valid

Coord ID• YYDDD Day of

Year• zzz

Arbitrary Extension

• External file name must match internal FILE TITLE

• When choosing zzz extension, user must be sure not to create a conflict with an existing file

12 January 2009


MiniMon Interrogation Files

• Multiple versions not supported– Files are maintained on server

– IDL provides insert capability only

• Format defined in SpecDoc?– Interrogation records YES

– Transceiver diagnosticsNO

• Varies by model / firmware

• FILE TITLE Record– Auto-generated by MiniMon

– sssYYDDD.an• sss Site Code

• YYDDD Day of Year

• a Sequence character

• n 1 or 2 (Primary/Backup)


12 January 2009














FishTag

Interrogation

12 January 2009














TimerTag

Interrogation

12 January 2009














TASSTransceiverDiagnostics

12 January 2009


Multimon Interrogation Files



• Format defined in SpecDoc?– Interrogation records NO




– sssYYDDD.nnn• sss Site

Code• YYDDD Day of

Year• nnn

Sequence number100’s primary, 200’s backup


12 January 2009











Year• nnn



DGEControlState

Events

12 January 2009











Year• nnn



DGEInterrogation

Events

12 January 2009











Year• nnn



TASSTransceiverDiagnostics

12 January 2009











Year• nnn



TimerTag

Interrogation

12 January 2009


Data Basics


• Tagging & Interrogation Files


12 January 2009


Core Database Tables

• Have existed from ‘day one’

• All contain data of interest to ‘typical user’ except …

– TEST_TAGS– TEST_TAG_DATA

• Many non-core tables exist to support PTAGIS processes

12 January 2009



These support IDL

12 January 2009







12 January 2009


PTTP – Client and Server Implementations

• Request processing– PTTP has processing, reporting, and messaging capabilities– What happens to ‘data payload’ depends on request and type of data

12 January 2009




Primary job

12 January 2009




• Request transformation– PTTP deployment began in Jan 2001

• Previously used upload or email for tagging files• Previously used upload or email for interrogation files• Deployment took time; backward compatibility was required

– Interrogation files emailed to [email protected] • Turned into PTTP transactions by PTTP_EmailClient• PTTP stages files for processing by IDL

– Tagging files transmitted by PTTP • Original email interface to FDVL was retained• PTTP_Dispatcher invokes PTTP_FDVL via PTTP_Handler • Sends email to [email protected] to invoke FDVL

Important secondary job

12 January 2009


PTTP Overview

PTTP ‘Transaction’ Zip-Files

• Name: TX<UUID>.zip

• Transmitted by FTP

• Unpacked in per-transaction directory– One serialized PTTP_Req (XML)– Zero or more data files

• Results available in PTTP_Ack (XML)

Remote PTTP Clients

• P3 - Tagging files

• Minimon, Multimon – Interrogation files

Local PTTP Clients

• PTTP_EmailClient– Generates PTTP transaction for

interrogation data sent by email to [email protected]

12 January 2009


PTTP Detail

Process flow …

1) watchFtp launches processPittagFtp

2) PTTP_Server unpacks zip-file to per-TX directory and instantiates PTTP_Req

3) PTTP_Server initializes PTTP_Ackand queues ‘Invocation’ event

4) PTTP_Server wakes PTTP_Dispatcherwhich proceeds according to event type

12 January 2009


PTTP Detail

Process flow …





12 January 2009


PTTP Detail

Process flow …





12 January 2009


PTTP Detail

Process flow …





PTTP_Req examines data payload. If interrogation files are included, the transaction identifier is posted to input queues of auxiliary analytical processes:

•TASS_Monitor

•DGE_Monitor

12 January 2009


PTTP Detail

Process flow …





For ‘Invocation’ event PTTP_Dispatcher …

• Invokes handler

• Handler reads possible Input file

• Handler performs action and queues‘Completion’ event

12 January 2009


PTTP Detail

Process flow …






• Invokes handler



File Type Handler

INTERROGATION PTTP_intIDL

MULTIMON PTTP_mulIDL

TAGGING PTTP_FDVL

12 January 2009


PTTP Detail

Process flow …






• Invokes handler



For ‘Completion’ event PTTP_Dispatcher

• Instantiates PTTP_Ack to log completion

• Calls genNextEvent() to queue possible follow-on ‘Invocation’ event

12 January 2009


PTTP Detail

Process flow …






• Invokes handler






12 January 2009


PTTP Detail

Process flow …






• Invokes handler






12 January 2009







12 January 2009


FDVL – Field Data Validating Loader

• Validation and Loading

• User Feedback– PTTP– Email

• Loader performance report

12 January 2009


FDVL – Field Data Validating Loader

• Validation and Loading• User Feedback

– PTTP– Email


12 January 2009


FDVL – Recording ‘Field Events’ in Database Tables

Original File Types

• Tagging

• ReleaseInformation

• MonitoredRelease

• Mortality

12 January 2009


FDVL – Recording ‘Field Events’ in Database Tables

Since 2001

• Tagging

• ReleaseInformation

• MonitoredRelease

• Mortality

Since 2001

Per-fish flag codes are usedto indicate recaptureand mortality eventswithin a Tagging file

12 January 2009


FDVL Overview

12 January 2009


FDVL Overview

PARALLEL

12 January 2009


FDVL Overview

SERIAL

12 January 2009


FDVL Overview

Staging eliminated with

fd_quickLoad

in August 2002

12 January 2009


FDVL Software Components

12 January 2009


FDVL - Meta-Data

• Stored and maintained in MS Access database

– spec_doc.mdb

• Exported to CSV-files for use– FD_FileMetaData.txt

– FD_FileTableMap.txt

• CSV-files populate in-memory data structures in fdc_receiveEmail.pl

– SpecDoc

– FileMap

12 January 2009


Specification Document: FDVL Model of PTAGIS File Types

12 January 2009


FDVL: Mapping of Record Fields to Columns in Database Tables

12 January 2009


FDVL: Modeling Content of a User’s File

12 January 2009


User’s Tagging File

• Must contain all mandatory records

• Values in optional fields must be valid

• As Validator parses file it – Assembles FileData tree

– Tallies ‘leaves’ on the tree

– Assesses their validity

– Errors exist if valid count is less than item count

• Processing stops with user message if errors exist

• Otherwise Loader is told to run

– May be running already

– May have been halted

12 January 2009


Validator Log File

For each record group• item_count

– Tally of required records

• valid_count– Tally of valid records

If errors are encountered• Specifics are posted to log• Log entries are incorporated

into user message

12 January 2009


Loader Log File

LOAD• Each body record loaded via

insert/update DB procedure• Record context determines

which proc is called– TAG_TAG

– TAG_REC

– TAG_MRT

• Duplicates encountered produce log message

– SelfDup or OtherDup

PURGE• Records unmarked by DB

proc are deleted

AGGREGATION• Record disposition by table

is tallied and logged

12 January 2009


Loader Log File

LOAD• Each body record loaded via

insert/update DB procedure• Record context determines

which proc is called– TAG_TAG

– TAG_REC

– TAG_MRT

• Duplicates encountered produce log message

– SelfDup or OtherDup

PURGE• Records unmarked by DB

proc are deleted

AGGREGATION• Record disposition by table

is tallied and logged

Determined by analysis of

flag codes in TAGGING file

BODY records

12 January 2009


FDVL Performance

• Performance report built to– Understand usage patterns

– Inform spreading of work away from heavy usage times if necessary

• Allows tracking performance week by week and year by year

– FDVL has always been able to outpace demand

– Loading rates have grown due to faster hardware

– Current average roughly 4000 tagging file ‘body’ records per minute

12 January 2009


FDVL Performance

• Performance report built to– Understand usage patterns

– Inform spreading of work away from heavy usage times if necessary

• Allows tracking performance week by week and year by year

– FDVL has always been able to outpace demand


– Current average roughly 4000 tagging file ‘body’ records per minute

Design has scaled well

12 January 2009


FDVL – Versioning of Submitted Files

• Database table: fd_file– Date and time of most recent submittal

– Submittal sequence number

– File archival directory (logical path name)

• Database table: fd_submit– Timestamps of arrival, validation, and loading

for each submittal

– Counts of body records by type

• item_count

• recap_count

• mort_count

• dot_out_count

• dup_count

12 January 2009







for each submittal


• item_count

• recap_count

• mort_count

• dot_out_count

• dup_count

Originally used for TAGGING files only

Now used for TAGGING and INTERROGATION files

12 January 2009







for each submittal


• item_count

• recap_count

• mort_count

• dot_out_count

• dup_count

Originally used for TAGGING files only

Now used for TAGGING and INTERROGATION files

Not applicable for INTERROGATION files

12 January 2009







12 January 2009


IDL – Interrogation Data Loader

• Validation and Loading• Loader performance report

• Interrogation System performance reports– TimerTag Report– TASS– CEA– DGE

12 January 2009


IDL – Loader Invocation

IDL is launched by

UNIX cron

‘every 3 hours’

12 January 2009



IDL is launched by

UNIX cron

‘every 3 hours’

We haven’t done thatfor a long time!

12 January 2009



PTTP

IDL is launched by

UNIX cron

‘every 3 hours’

12 January 2009



PTTP

IDL is launched by

UNIX cron

‘every 3 hours’

Files are pre-processed byidl_validateFile.pl

one-by-one as IDL runs

12 January 2009



These support IDL

12 January 2009


IDL – Database Tables

12 January 2009


IDL – Operation in Serial Mode

Output from idl_validateFile.pl

12 January 2009


IDL – Operation in Serial Mode

Output from idl_validateFile.pl

Logic is contained in load_int_data_cdbp

‘called’ database procedure

12 January 2009


IDL – Motivation for Parallel Loading Capability

12 January 2009


IDL – Primary Storage Structure For Parallel Operation

12 January 2009


IDL – Operation in Parallel Mode

PTTP

12 January 2009



Partition the Data Stream

12 January 2009



Drop Indexes

12 January 2009



Load Data

12 January 2009



Build Indexes

12 January 2009




• Loader performance report• Interrogation System performance reports

– TimerTag Report– TASS– CEA– DGE

12 January 2009


IDL Performance

• Performance report allows year-to-year comparisons


– Overall IDL run times have decreased even though data volumes have increased

• With small data sets – Per-file overhead

dominates

• With large data sets– Most time actually spent

loading data– Average loading rates

exceed 1500 rows/min

• Haven’t resorted to Parallel mode for many years

– It’s our ‘ace in the hole’ if we need to scale further

12 January 2009


IDL Performance

• Performance report allows year-to-year comparisons


– Overall IDL run times have decreased even though data volumes have increased

• With small data sets – Per-file overhead

dominates

• With large data sets– Most time actually spent

loading data– Average loading rates

exceed 1500 rows/min

• Haven’t resorted to Parallel mode for many years

– It’s our ‘ace in the hole’ if we need to scale further

Design has scaled well

12 January 2009






12 January 2009






12 January 2009


Type of timer-tag depends on transceiver …

FS1001 Physical timer-tag at each coil FS1001A Virtual timer-tag in transceiver

FS1001M Virtual timer-tag in transceiver

Timer Tag

- Known PIT-code emitted periodically to signal that coil and transceiver are working properly

Transceiver

Coil

Physical Timer Tag- Actual tag mounted close to coil- Normally shunted to ground- Transceiver un-grounds to test- More likely detected than actual fish

Virtual Timer Tag- Code generated by transceiver circuitry- Transceiver contains antenna emulator- Code is presented to emulator to test- Likelihood of detection similar to actual fish

12 January 2009


TimerTag Performance Report

• Launched by UNIX cron after IDL runs

• Red cells indicate missing TimerTag detections

• Black vertical bars (none in this example) indicate inter-file ‘data gaps’

• HTML links in Extension row display associated interrogation file in browser

12 January 2009






12 January 2009


TASSTransceiver Annunciator / Statistical Summarizer

• Alarms displayed as “red light” – Site annunciator - SLC500 comms failure– Coil annunciator - Timer-tag failure

• Corrective actions posted to event log

• Transceiver diagnostics reported– Basic: Alarms and parameter values– Enhanced: Noise and temperature plots vs. time

12 January 2009


12 January 2009


- ReportingPolicy determines what is displayed- Established in configuration file- Any number of Annunciator Panels can be defined

12 January 2009


12 January 2009




• Corrective actions posted to event log• Transceiver diagnostics reported

– Basic: Alarms and parameter values– Enhanced: Noise and temperature plots vs. time

12 January 2009


12 January 2009


Text entered here is posted to the site event-log forum and text file.

Sets lights green

Displays report

Displays file

12 January 2009





• Transceiver diagnostics reported– Basic: Alarms and parameter values– Enhanced: Noise and temperature plots vs. time

12 January 2009


Three types of transceiver currently in use

FS1001 Alarm & parameter reporting FS1001A Enhanced diagnostics (noise)

FS1001M Multiple “coils”

Transceiver

- Transmitter / Receiver- Decodes Tag-ID from digital signal- Periodically energizes TimerTag

Transceiver - “Unique” ON or OFF

Coil

Single-Coil

Multiplexed

12 January 2009


Three types of transceiver currently in use

FS1001 Alarm & parameter reporting FS1001A Enhanced diagnostics (noise)

FS1001M Multiple “coils”

Transceiver

- Transmitter / Receiver- Decodes Tag-ID from digital signal- Periodically energizes TimerTag

Transceiver - “Unique” ON or OFF

Coil

Single-Coil

Multiplexed

Content and format of diagnostic messages written to data stream

depend on transceiver model

12 January 2009





• Transceiver diagnostics reported

– Basic: Alarms and parameter values– Enhanced: Noise and temperature plots vs. time

12 January 2009


12 January 2009


Actual time-stamp value

“Almost-same” time-stamp; actual times differ

12 January 2009





• Transceiver diagnostics reported– Basic: Alarms and parameter values

– Enhanced: Noise and temperature plots vs. time

12 January 2009


Min MaxMean

+/- 3-Sigma

12 January 2009


Min MaxMean

+/- 3-Sigma

Click the light for a particular coil

12 January 2009






Parsing & Statistics

12 January 2009






DataStructures & Algorithms

12 January 2009






DataStructures & Algorithms

Hydraulic Topology

Specifically: “How fish move through the facility”

12 January 2009


“OK, it’s an interrogation site …”

JUVENILE?

ADULT?

… but there are many differences!!

Transceiver Model

FS1001FS1001AFS1001M

Adult and Juvenile sites have this in

common

How Could We Describe An INTERROGATION Site?

12 January 2009



JUVENILE?

ADULT?


Transceiver Model



common


What Else?

12 January 2009


Attributes in Common: Adult & Juvenile Facilities

ADULT JUVENILE

Same technology is used for detecting fish: transceiver_id One or more transceivers transceiver_mfgr Same transceiver makes transceiver_model Same range of transceiver models antenna_id One or more antennas per transceiver

Water flows downstream through both kinds of site: flow_node_id Water enters and leaves site through one or more “nodes” node_type For example: Weir, SlideGate flow_path_id Water follows “paths” between “nodes” rf_energy_where “Where the fish are!”: WeirOrifice, ClosedPipe, Flume

12 January 2009



JUVENILE?

ADULT?


Transceiver Model



common


What Differences?

12 January 2009


Attributes: Juvenile Fish Bypass Facility

Only juvenile fish, all travelling downstream: node_type For example: FishSeparator, SlideGate

Fish are entrained in flow - Movement through “monitors” is “continuous”: continuous_flow_y_n “Y” - Yes, what goes in must come out!

12 January 2009


Attributes: Adult Fish Ladder

Mix of adult and juvenile fish, travelling both upstream and downstream: node_type For example: OverflowWeir, SlottedWeir, CountingWindow

Fish are not entrained in flow - Movement through “monitors” is “discontinuous”: continuous_flow_y_n “N” - No, a fish can turn around and retrace its path.

12 January 2009


Attributes: Adult Fish Ladder

Mix of adult and juvenile fish, travelling both upstream and downstream: node_type For example: OverflowWeir, SlottedWeir, CountingWindow

Fish are not entrained in flow - Movement through “monitors” is “discontinuous”: continuous_flow_y_n “N” - No, a fish can turn around and retrace its path.

Or ‘in-stream’ site

12 January 2009


Existing SiteConfigurationManagement Schema

What if: Activity-TechnicalConfiguration Schema

antenna_group monitor

antenna site_coilATC Schema (strawman)Activity-TechnicalConfiguration Schema

12 January 2009






Cumulative Detection Efficiency Analyzer

12 January 2009


Terminology - Monitor

Interrogation Monitor

- Two or more “coils” (antennas)- Flow continuity- High probability of detecting fish

Transceiver Energizes the coil

CoilEnergizes PIT-tags as

they pass through

12 January 2009


CEA Probability of Missing Fish is valid only where …

Interrogation Monitor

- Two or more “coils” (antennas)- Flow continuity- High probability of detecting fish

Transceiver Energizes the coil

CoilEnergizes PIT-tags as

they pass through

Fish are entrained in flow

12 January 2009


CEA Textual Report

• Launched by UNIX cron– After each IDL run

– Once per week for aggregation

• Provides quick overview of interrogation activity at a site

• Count of fish seen and missed by coil at each monitor

• Probability of missing fish at each monitor

• Daily and by-week aggregations – Stored in database

– Accessible from web site

12 January 2009


CEA Textual Report

• Launched by UNIX cron– after each IDL run

– once per week for aggregation

• Provides quick overview of interrogation activity at a site

• Count of fish seen and missed by coil at each monitor

• Probability of missing fish at each monitor

• Daily and by-week aggregations – Stored in database

– Accessible from web site

Depends on flow continuity

12 January 2009


CEA Web Report

• Web interface gives easy access to current and historical results

• Report is useful as a simple indicator of fish activity

• For this reason, results are produced even for sites where flow continuity does not exist

• FishSeen link explains …

12 January 2009


12 January 2009


Visio file • UML ComponentDiagram (shown)• UML StaticStructure• UML UseCase

Textual documentation on ‘sockeye’ • /home/ptagdev/ptagis3/source/cum_det_eff

«executable»cea_processYesterdayWeekly.sh

«executable»cea_processYesterdayDaily.sh

«executable»p3_daily_4.sh

«call» «executable»env_common_defs.sh

«executable»env_ptagis_master.sh

The ‘YESTERDAY’ environment variable is defined here

«executable»CEA_processHistoricalData.pl

«executable»p3_daily_3.sh

«executable»p3_weekly_3.sh

«executable»CEA_Direct_weekly.sh

«executable»CEA_Convert_Weekly.sh

«executable»CEA_Direct_daily.sh

«executable»CEA_Convert_Daily.sh

«call»«call»

«import»

$CEA_REP_DIR/

«executable»run_cea_direct.pl

«executable»cea_direct.pl

ALL_weekly.txt ALL_daily.txt

Produce one output file for each active interrogation site.

Concatenate per-site filesto produce the “ALL” file

«executable»CEA_convertA2S.pl

«executable»CEA_reportHistoricalData.pl

«executable»CEA_reportYTDData.pl

«table»PTAGIS3_InterrogationData

«table»PTAGIS3_CumulativeEffData

«executable»CEA_generateReport.pl

$CEA_ARCH_DIR/

$CEA_PUB_FTP_DIR/

NOTE: ‘weekly’ and ‘2-week’ invocations are no-op unless YESTERDAY is a Sunday

CEA_AST-SQL: Component DiagramDoug Clough, SYNERGETICSVer. 1.0 21 Nov 2003

CEA_AST-SQL: Component Diagram

ReportWriter files and outputs

CSV output files

CEA DocumentationCumulative Efficiency Analyzer

12 January 2009






12 January 2009


DGEDiversion Gate Efficiency Analyzer

• DGE Configuration File basics• How DGE replays real-world events

• Last-known PLC values– Sample rates (Data > Reports > SystemGen > DGE PLC)

– Switch settings (Farther down the same page)

• Last 24-hours & YTD efficiencies– Action summary by control context– Gate actions - success & failure - by hour

12 January 2009


DGE Configuration File Basics

Multimon’s SxC implementation

• Needs to know what gate is controlled by a given monitor– This information derives from facility ‘port and transceiver’ layout

• Does not need to know– Where a diverted fish is seen subsequent to diversion– What detections subsequent to diversion mean with respect to

success or failure of diversion attempt

DGE

• Needs to know what gate is controlled by a given monitor

• Also needs to know– Where a diverted fish is seen subsequent to diversion– What detections subsequent to diversion mean with respect to

success or failure of diversion attempt– This information derives from facility ‘concrete and steel’ layout

which is not represented in the database– DGE’s site configuration files provide this information

12 January 2009







DGE





Network Topology

Hydraulic Topology

12 January 2009







DGE





Network Topology

M4 – If it’s to provide this functionality …

Hydraulic Topology

Both Are Needed

12 January 2009


Little Goose Smolt Bypass and Collection (GOJ)

12 January 2009


http://www.psmfc.org/pittag/DGE/persist/2004/GOJ/GOJ_Config.txt

12 January 2009


AST-formatted ASCII file:

- Indentation captures hierarchical relationships- Designed as an “outline” to be read by people- Think of it as XML without tags

persistenceHashAssemblage

file

12 January 2009

Pacific States Marine Fisheries Commission http://www.psmfc.org/pittag/DGE/persist/2004/GOJ/GOJ_Config.txt

12 January 2009

Pacific States Marine Fisheries Commission http://www.psmfc.org/pittag/DGE/persist/2004/GOJ/GOJ_Config.txt

Variable names

Site topology

List of Gates controlled by this Monitor

12 January 2009


Variable names

Variable-name Mapping

User-friendly

Internal global

Site topology

List of Gates controlled by this Monitor

12 January 2009


Site topology

Gate behavior

12 January 2009


Internal globalInternal

local

Site topology

Gate behavior

ControlList PLC switch settings that affect this gate

12 January 2009


Site topology

Gate behavior

What to do in a given

ControlContext

ControlContext Set of ControlList values at any given time

12 January 2009



• DGE Configuration File basics

• How DGE replays real-world events• Last-known PLC values

– Sample rates (Data > Reports > SystemGen > DGE PLC)



12 January 2009


PTTP Drives DGE

• $DGE_LOG serves as input queue for DGE_Monitor

• Arrival of $TX_ID in queue wakes up DGE_Monitor

• If $TX_ID contains files from sites in SiteList, launch DGE_ProcessFile

• DGE Process Overview1) Instantiate site model

2) Process available files

3) When queue is empty, persist component objects

4) Launch DGE_Driver_Reports

$PTTP_SERVER_PERSIST_DIR

Several 'Mini/Multimon' files from one Site transmitted in a single PTTP 'zip-file'

DGE_Monitor.pl

DGE_processFile

.pl

DGE Archives for Site

"volatilize" "persist"

InterrogationSite PTTP: <TX_ID>.zip

Per-$TX_ID Directory

$DGE_LOG

PTTP_Req

$TX_ID

"tail -f"

"Un-zip"

SiteList

$MULTIMON_DIR

Store each file

File listINT or MUL

DGE_Driver_Reports.pl

R_AST

1

2

12 January 2009


DGE Process

1) Instantiate site model from configuration file and per-component persistence files

2) Retrieve ‘events’ from interrogation file

3) Delegate ‘control state’ events to PLC object

4) Propagate ‘detection events’ throughout gate-monitor ‘flow paths’

5) Register diversion successes and failures

6) Persist until next time

DGE_Monitor

DGE_processFile.pl

$MULTIMON_DATA

$DGE_PERSIST_DIR

.../<SiteCode>

DGE_Detection

Facility

FileList

<SiteCode>YYJJJ.nnn

DGE_Component

<SiteCode>_Config.txt

DGE_MultiCoilDetector

DGE_Gate

DGE_Prog

Controller

DGE_SlideGate

DGE_2WayGate

DGE_3WayGate

As dictated by Config file

<SiteCode>

Loop through the file

DGE_Event

DGE_Detection

Event

DGE_ControlState

Event

<SiteCode>.archive

Instantiate

Persist

1

2

3

12 January 2009


Read config file and instantiate site model

Set PLC variables from persistence file

Establish relationships between site components

12 January 2009


Sets a PLC variable

Read interrogation file and generate DGE_Events

ControlState_Event

12 January 2009


Interrogation_Event at Monitor downstream from DiversionGate

Tag-ID Coil-ID

[DiversionRequest]

12 January 2009


DGE_Component instances propagate Interrogation_Event

upstream

12 January 2009



upstream

12 January 2009



upstream

12 January 2009


Interrogation_Event at Monitor that controls a DiversionGate

12 January 2009




upstream

DGE_Monitor instance forwards Interrogation_Event to

associated DiversionGate(s)

12 January 2009




upstream



12 January 2009


Diversion_Gate retrieves ControlContext from PLC



upstream



12 January 2009




upstream



ControlContext and DiversionRequest determine required GateAction

12 January 2009



12 January 2009




upstream

12 January 2009




upstream

Since the Tag-ID was seen here, the gate

was CLOSED

DiversionGate remembers requested GateAction; compares with actual

12 January 2009




upstream

DiversionGate tallies actions by ControlContext:Request:Actual

Since the Tag-ID was seen here, the gate

was CLOSED

12 January 2009




• How DGE replays real-world events




12 January 2009


12 January 2009




• How DGE replays real-world events




12 January 2009


12 January 2009


12 January 2009


12 January 2009


12 January 2009






12 January 2009






There have been others!!

12 January 2009


DEA – 1995-?Instantaneous Detection Efficiency Analyzer

• Developed when 400khz tags were in use– Transceivers provided no diagnostic messages– Statistical analysis gave detection efficiency estimate

• Ring-buffer data structure maintained interrogation context– Multiple interrogations of a single tag still counted as one fish– Computed probability based on ‘tally groups’ of 20 fish

• Not used much since advent of ‘advanced’ transceivers– Self-configuring; costs nothing to let it run

12 January 2009


• Fish not entrained in water flow– Adult fish can jump over weirs– A single fish can traverse weir upstream and downstream

• Ring-buffer data structure maintained interrogation context– Multiple interrogations of a single tag still counted as one fish– Computed probability based on ‘tally groups’ of 20 fish

• Retired when counting windows were instrumented– Provided ‘close enough’ approximation of flow continuity

ADE – 2001-2008Adult Detection Efficiency Analyzer

12 January 2009







12 January 2009


Concluding Remarks

• PTAGIS continually evolving

• Development of server-side tools driven by evolution of – PIT tag technology– Transceiver capabilities

• Stable processing environment has maximized time available for addressing needs of

– Researchers– PTOC O&M Staff

• Major factors providing stability– UNIX (POSIX) standards– Language and API ‘interface stability’ of Perl and CPAN modules– Ingres RDBMS

• Practically ‘hands-off’ day by day operation

• Continued evolution of VLDB features

• Good Tech Support

OVERVIEW OF SERVER-SIDE PROCESSING Doug Clough SYNERGETICS Engineered Systems 12 January 2009 .

Documents