Introduction and Detailed System Overview: Science Data … · Introduction and Detailed System Overview: Science Data Processing Internal Training Introduction and Detailed System

625-EMD-002, Rev. 02

Introduction and Detailed System Overview: Science Data Processing Internal Training

Introduction and Detailed System Overview: Introduction and Detailed System Overview: Science Data Processing Internal TrainingScience Data Processing Internal Training

July 2006July 2006

EMD SDP Internal TrainingEMD SDP Internal Training

INSE

RTIO

NRETRIEVAL

PRODUCTION

SubscribeSearch& Order

Store

ExternalData Provider User

Deliver

Generate Plan

Ingest

User

625-EMD-002, Rev. 02

3625-EMD-002, Rev. 02

Objectives

• Overall objective: Describe ECS structure and function for Science Data Processing (SDP)– Identify subsystems and Computer Software Configuration

Items (CSCIs)– Specify major components and functions/processes of CSCIs– Describe role of CSCIs/functions/processes in the context of

ECS operational scenarios- ASTER-specific functions (e.g., DAR, expedited data support)- Producing and distributing data products (including media)- Updating QA metadata- On-demand processing- User registration- MODIS data access

4625-EMD-002, Rev. 02

What This Lesson Is (and Is Not)• Is

– Brief illustration of ECS high-level structure– Introduction to subsystems that make up ECS at a site– Examination of each subsystem and its Computer Software

Configuration Items (CSCIs), with components- Introduction of all system elements and brief description of functions- Background for subsequent scenario-based presentation of system

functional flows– Detailed look at system functioning in the context of operational

scenarios• Is Not

– Full description of overall ECS structure and function– Description of specific individual ECS entities (e.g., SMC)– Software development lesson– Complete description of interfaces and event sequences– Operations training

5625-EMD-002, Rev. 02

Program Overview

NASAScience Mission Directorate

U.S. Global Change Research Program

Earth Observing System

Terra (AM)Aqua (PM)AURA(CHEM)

Other

Earth Probes &Focused Missions

UARSTOPEXTRMMTOMSOther

EDOSEMSnECS

Science Satellites EOSDISEOS:- Designed to collectEarth Science Data

- 3 Components: spacesystem, ground system,research system.

EOSDIS

460 GB/DayRaw Data

28 GB/DayRaw Data

EOSDIS:- Provides the ground system for

collection and analysis of science data

Science Data User

1600 GB/DayProcessed Data

Science Data User:- Science Researchers

Worldwide Global Change Research Program

6625-EMD-002, Rev. 02

EOSDIS Principal Components

White Sands

S/C DataAcquisition

TDRSS

Complex

Polar Stations

ECSDataCommunications

EOSMissionSupportNetwork(EMSn)

KEYECSEDOSData Comm

S/COperations

EOS Spacecraft

USERS

USERS

USERS

USERS

Distribution/Access

DataCommunications

EOS Science

NetworkSupport

(ESSn)

EDOS

EOS Dataand

OperationsSystem (EDOS)

EOS Real-time Processing

System (ERPS)

ECS Mission OperationsSystem (EMOS)

Science Data ProcessingSegment (SDPS)

Communications andSystem Management Segment

(CSMS)

Instrument Support Toolkits

EOSDIS

Flight Dynamics

7625-EMD-002, Rev. 02

EOSDIS Data Flow

Data from other Mission Data Systems(TRMM, ICESat, SAGE III, . . .)

ECS MissionOperations

System(EMOS)

EOS

Miss

ion

Supp

ort N

etwo

rk (E

MSn)EDOS

FlightDynamics

System

Totally suppliedby ECSContains ECScomponents

ScienceComputingFacilities

ScienceCommunity

CooperatingData Centers

InstrumentSupport Toolkit

PI/TL Facility

EOS Science Support Network (ESSn)

InternationalPartner

Ground Systems, Operations Centers

Data ScienceInvestigator-

led ProcessingSystems (SIPS)

ECS DAACsGES ASDC NSIDC LP.DAAC

ScienceData

Processing

Non-ECS DAACsASF SEDAC ORNL

Communications andInternetworking Infrastructure

Non-ECS DAACsGES ASDCNSIDC LP.DAAC

PO.DAAC

InternationalPartner

Data Centers

8625-EMD-002, Rev. 02

ECS SDP Context

Insertand

Retrieve

Data Server Subsystem (DSS)

SDSRVDDIST

STMGT

Inven-toryDB

Collection

Archive

Communications (CSS)and Internetworking (ISS)

EMOS

ExternalData Sourcesand Providers

ScienceSoftwareand Data

IngestSubsystem

(INS) Ingest

Data

PUSH

InventorySearch

Data ProcessingSubsystem

(DPS) Data Proc

0110001011001011001011010111010110001011

Data ProcessingRequests

PlanningSubsystem

(PLS) Planning

abc

def

ghi

jkl

mno

PULL

External/InternalUsers

Acquire

User Registration, OrderStatus, User Profile

Client (CLS)External/InternalUsers

System MgmtSubsystem

MSS

Hard MediaOrders

ProductDistributionSystem (PDS)

AcquireRequests

Data/Services

ProductRequests

ProductOrders

Order ManagerSubsystem

(OMS)ElectronicProducts

Data

Hard MediaProducts

Searchand

Download

Data Pool(DPL)

DATAPOOL

Searchand

Access

Data MgmtSubsystem

(DMS)

Meta-data

Data

Spatial Subscrip-tion Server

(SSS)

9625-EMD-002, Rev. 02

Subsystems and Functions

Science Data Processing Segment (SDPS)• Data Server Subsystem (DSS)

– Data storage and management: archive science data (with related insert, search and retrieve functions), archive management, data resource staging

• Product Distribution System (PDS)– Service for hard media orders, in conjunction with DSS and

Order Manager Server (OMS)• Ingest Subsystem (INS)

– Interface with external data providers and transfer data into ECS (with related staging functions and operator interfaces)

• Spatial Subscription Server (SSS)– Creation and management of subscriptions for data

distribution/notification and for Data Pool insert

10625-EMD-002, Rev. 02

Subsystems and Functions (Cont.)

SDPS (Cont.)• Data Pool (DPL)

– Provides on-line access for browsing and FTP download of selected granules, metadata, and browse data

• Client Subsystem (CLS)– Provides interfaces and access for external users

• Data Management Subsystem (DMS)– Enables cross-site data search and retrieval; gateway for

interface of ECS with EOS Data Gateway Web Client (Version 0 IMS) protocol

• Order Manager Subsystem (OMS)– Manages orders from EDG and other sources, distributing them

to appropriate ECS services (SDSRV, PDS)

11625-EMD-002, Rev. 02


SDPS (Cont.)

• Planning Subsystem (PLS)– Long- and short-term planning of science data processing, and

management of production resources• Data Processing Subsystem (DPS)

– Dispatches and monitors execution of science software

12625-EMD-002, Rev. 02


Communications and System Management Segment (CSMS)• System Management Subsystem (MSS)

– System maintenance, management, and administration (includes trouble ticketing, baseline and configuration management, fault and performance monitoring, and user account management and order management)

• Communications Subsystem (CSS)– General system infrastructure functions (includes network

communications, libraries to standardize software mechanisms, application error handling, subscription service, interfaces to e-mail, file transfer and file copy)

• Internetworking Subsystem (ISS)– Networking hardware devices and embedded softwareNOTE: The ISS is part of the ECS infrastructure and is not

addressed in detail in this course.

13625-EMD-002, Rev. 02


DataData Pool

(DPL)DATAPOOL

ElectronicProducts


(DPS) Data Proc

0110001011001011001011010111010110001011

InventorySearch


Archive


SDSRVDDIST

STMGT

Inven-toryDB

Collection

Insertand

Retrieve


PUSH


PlanningSubsystem

(PLS) Planning

abc

def

ghi

jkl

mno

PULLSystem Mgmt

SubsystemMSS

AcquireRequests

ProductOrders


(OMS)Hard MediaOrders

Meta-data

ProductRequests

EMOS


Searchand

Access


Data MgmtSubsystem

(DMS)


Searchand

Download



(SSS)Hard MediaProducts

AcquireData/Services

Data

DataIngest

Subsystem(INS) Ingest

Subsystems and CSCIs: DSS


SDSRVDDIST

STMGT

Inven-toryDB

Collection

14625-EMD-002, Rev. 02

Subsystems and CSCIs: DSS (Cont.)

• Data Server Subsystem (DSS)– Provides capabilities to store, search,

retrieve, and distribute earth scienceand related data

– Client-server information transfer is by commands and requests

– Generates Universal References to identify ECS entities- Granule UR: represents a granule in the data server (e.g., as

follows)UR:10:DsShESDTUR:UR:15:DsShSciServerUR:13:[GSF:DSSDSRV]:16:SC:MOD10_L2:1411

- Server UR: represents a specific running data server application (e.g., )

– Interfaces with virtually all ECS subsystems and components– Uses several COTS tools: RogueWave tools and libraries,

Sybase relational database, Spatial Query Server

Data Server Subsystem

SDSRVDDIST

STMGT

Inven-toryDB

Collection

DsShSciServerUR

A granule is thesmallest piece ofdata that isindependentlymanaged by thesystem, i.e.,represented by arecord in the inventory.

15625-EMD-002, Rev. 02


• Science Data Server (SDSRV) CSCI– Provides the ECS with a catalog of Earth

Science Data holdings, and the Earth ScienceData Type (ESDT) services that operate on the data

– Manages and provides user access to data collections through its catalog of metadata and mechanisms to acquire data from the archive

– Seven major components- Science Data Server - services requests for storage, search,

retrieval, and manipulation of science data- HDF EOS Server - provides science data subsetting- Science Data Server GUI - provides operator interface- Granule Deletion Tool - provides a command-line interface for

deleting granules- Science Data Server Command Line Interface (SCLI) - provides

interface to the S4PM processing system and the Product Distribution System (PDS)

SDSRVInven-toryDB

16625-EMD-002, Rev. 02


• Science Data Server (SDSRV) CSCI (Cont.)– Seven major components (Cont.)

- Sybase ASE Server - manages catalog (metadata)- SQS Server - manages catalog (specialized spatial searches)

17625-EMD-002, Rev. 02

EcDsScienceDataServer

EcDsSdSrvGui

ASE (Sybase)

ECS InventoryDatabase

EcDsHdfEosServer

Spatial QueryServer (SQS)

EcDsBulkDelete.pl

EcDsSCLI

Subsystems and CSCIs: DSS SDSRV Architecture and Interfaces

18625-EMD-002, Rev. 02


EcDsSdSrvGui

ASE (Sybase)


EcDsHdfEosServer


EcDsBulkDelete.pl

EcDsSCLI


Metadata/Spatial Metadata storage request;Query

Metadata/Spatial Metadata retrieval

19625-EMD-002, Rev. 02


EcDsSdSrvGui

ASE (Sybase)


EcDsHdfEosServer


EcDsBulkDelete.pl

EcDsSCLI


Metadata/storage request;Query

Metadata/retrieval

20625-EMD-002, Rev. 02


EcDsSdSrvGui

ASE (Sybase)


EcDsHdfEosServer


EcDsBulkDelete.pl

EcDsSCLI


Request On-Demand

Granule Deletion

21625-EMD-002, Rev. 02


EcDsSdSrvGui

ASE (Sybase)


EcDsHdfEosServer


EcDsBulkDelete.pl

EcDsSCLI


System status requests and controls; ESDT install/update requests

Error and system status information

22625-EMD-002, Rev. 02


EcDsSdSrvGui

ASE (Sybase)


EcDsHdfEosServer


EcDsBulkDelete.pl

EcDsSCLI


Distribution Request;

Push Data Instructions;

Acquire Subsetted Data

23625-EMD-002, Rev. 02


EcDsSdSrvGui

ASE (Sybase)


EcDsHdfEosServer


EcDsBulkDelete.pl

EcDsSCLI


Subsetting request;

Reformatting request

Subsetting status

24625-EMD-002, Rev. 02


EcDsSdSrvGui

ASE (Sybase)


EcDsHdfEosServer


EcDsBulkDelete.pl

EcDsSCLI


Staging disk creation; archive requests

25625-EMD-002, Rev. 02


EcDsSdSrvGui

ASE (Sybase)


EcDsHdfEosServer


EcDsBulkDelete.pl

EcDsSCLI


Inventory search

requests

ESDT installation

information; replace

ESDT installation

information

26625-EMD-002, Rev. 02


EcDsSdSrvGui

ASE (Sybase)


EcDsHdfEosServer


EcDsBulkDelete.pl

EcDsSCLI


Granule value

parameter requests;

submit user a

cquire

(EcPlOdMgr)

Granule value

parameters (metadata

attributes)

27625-EMD-002, Rev. 02


EcDsSdSrvGui

ASE (Sybase)


EcDsHdfEosServer


EcDsBulkDelete.pl

EcDsSCLI


MCF requests;

granule

acquire/deletio

n

requests; data

insert requests;

metadata update

requests

Metadata configuration

files (MCFs); product

history

28625-EMD-002, Rev. 02


EcDsSdSrvGui

ASE (Sybase)


EcDsHdfEosServer


EcDsBulkDelete.pl

EcDsSCLI


Event notifica-fication; URs of inserted granules; register/replace

events; request distribution media options

Acquire requests based on subscriptions; request info for Expedited Data Set Notification (EDN); search; media options

29625-EMD-002, Rev. 02


EcDsSdSrvGui

ASE (Sybase)


EcDsHdfEosServer


EcDsBulkDelete.pl

EcDsSCLI


Requests for metadata config-

uration files and validation of

metadata; data insertion requests

Metadata configura-tion files; metadata verification

30625-EMD-002, Rev. 02


EcDsSdSrvGui

ASE (Sybase)


EcDsHdfEosServer


EcDsBulkDelete.pl

EcDsSCLI


User Profile

Request User Profile

31625-EMD-002, Rev. 02


EcDsSdSrvGui

ASE (Sybase)


EcDsHdfEosServer


EcDsBulkDelete.pl

EcDsSCLI


Non-Synergy IV: Product

Requests for non-media

distribution

32625-EMD-002, Rev. 02


EcDsSdSrvGui

ASE (Sybase)


EcDsHdfEosServer


EcDsBulkDelete.pl

EcDsSCLI


Non-Synergy IV: Product Requests for data to fill media orders

33625-EMD-002, Rev. 02


EcDsSdSrvGui

ASE (Sybase)


EcDsHdfEosServer


EcDsBulkDelete.pl

EcDsSCLI


Non-Synergy IV: Query forgranule size

Non-Synergy IV: Granule size

34625-EMD-002, Rev. 02


EcDsSdSrvGui

ASE (Sybase)


EcDsHdfEosServer


EcDsBulkDelete.pl

EcDsSCLI


Request metadata

Metadata for conversion and storage in Data Pool

35625-EMD-002, Rev. 02


• Storage Management (STMGT) CSCI– Stores, manages, and retrieves data files on behalf of other

science data processing components– Six major components

- Archive Server - provides GUI and access to stored data- Cache/Staging Manager - Cache Manager server and Staging Disk

server manage data files that have been retrieved from the archive and placed into a cache area on staging disk

- Media Server Process - schedules access to shared peripheral resources (FTP, secure copy) and devices for Ingest (8mm, DTF-2)

- Pull Monitor - links to Cache Manager to manage files in the user pull area, making them eligible for deletion as they are retrieved by users or as their time-out periods expire

- Request Manager - routes requests from clients to servers- Data Base - contains data tables for STMGT devices, cache

management, event and log management, requests, and related functions

STMGT

36625-EMD-002, Rev. 02

Subsystems and CSCIs: DSS STMGT Architecture and Interfaces

ASE (Sybase) STMGT/DDISTDatabase

AMASS

DiskIndexFile

StagingDataList

Device InfoRequest List

EcDsStArchiveServer EcDsStCacheManagerServer

EcDsSt8MMServerEcDsStDTFServer

EcDsStFtpServer

EcDsStPullMonitorServer

EcDsStmgtGui

EcDsStStagingDiskServer

FTP Daemon

ResourceList

EcDsStRequestManagerServerEcDsStCopyServer

37625-EMD-002, Rev. 02


AMASS

DiskIndexFile

StagingDataList




EcDsStFtpServer


EcDsStmgtGui


FTP Daemon

ResourceList



Send all requests, for routing

Ingest Function

38625-EMD-002, Rev. 02


AMASS

DiskIndexFile

StagingDataList




EcDsStFtpServer


EcDsStmgtGui


FTP Daemon

ResourceList



Ingest Function

Create_Allocate;

Destroy_Deallocate

39625-EMD-002, Rev. 02


AMASS

DiskIndexFile

StagingDataList




EcDsStFtpServer


EcDsStmgtGui


FTP Daemon

ResourceList



Ingest Function

Allocate device; Mount

tape; Ingest to; Unmount

tape; Deallocate device;

Create_Allocate; Destroy_Deallocate

40625-EMD-002, Rev. 02


AMASS

DiskIndexFile

StagingDataList




EcDsStFtpServer


EcDsStmgtGui


FTP Daemon

ResourceList



Transfer files

Ingest Function

Implement secure file transfers

41625-EMD-002, Rev. 02


AMASS

DiskIndexFile

StagingDataList




EcDsStFtpServer


EcDsStmgtGui


FTP Daemon

ResourceList



Ingest Function

Device management and staging disk attachment information

Staging disk

attachment

information

Staging disk management information

42625-EMD-002, Rev. 02


AMASS

DiskIndexFile

StagingDataList




EcDsStFtpServer


EcDsStmgtGui


FTP Daemon

ResourceList



Ingest Function

Request Data Insert

43625-EMD-002, Rev. 02


AMASS

DiskIndexFile

StagingDataList




EcDsStFtpServer


EcDsStmgtGui


FTP Daemon

ResourceList



Ingest Function

Copy granule; copy browse

44625-EMD-002, Rev. 02


AMASS

DiskIndexFile

StagingDataList




EcDsStFtpServer


EcDsStmgtGui


FTP Daemon

ResourceList



Ingest Function

Staging disk

attachment informa-

tion; Get volume

group/server/path

mapping

Volume group/server/ path mapping used to store data for the specified ESDT

45625-EMD-002, Rev. 02


AMASS

DiskIndexFile

StagingDataList




EcDsStFtpServer


EcDsStmgtGui


FTP Daemon

ResourceList



Ingest Function

Status monitoring and control

46625-EMD-002, Rev. 02


AMASS

DiskIndexFile

StagingDataList




EcDsStFtpServer


EcDsStmgtGui


FTP Daemon

ResourceList




DistributionStagingFunction

47625-EMD-002, Rev. 02


AMASS

DiskIndexFile

StagingDataList




EcDsStFtpServer


EcDsStmgtGui


FTP Daemon

ResourceList




Create_Allocate;Destroy_Detach;Link files

48625-EMD-002, Rev. 02


AMASS

DiskIndexFile

StagingDataList




EcDsStFtpServer


EcDsStmgtGui


FTP Daemon

ResourceList




Retrieve data;

Create_Attach/Allocate;

Destroy_Detach/Deallocate;

claim ownership; link file

49625-EMD-002, Rev. 02


AMASS

DiskIndexFile

StagingDataList




EcDsStFtpServer


EcDsStmgtGui


FTP Daemon

ResourceList




Create_Attach/Allocate;

Destroy_Detach/Deallocate;

link file

50625-EMD-002, Rev. 02


AMASS

DiskIndexFile

StagingDataList




EcDsStFtpServer


EcDsStmgtGui


FTP Daemon

ResourceList




Get volume group/server/path mapping

Volume group/server/path mapping

Retrieve data via

volume

group/server/path

51625-EMD-002, Rev. 02


AMASS

DiskIndexFile

StagingDataList




EcDsStFtpServer


EcDsStmgtGui


FTP Daemon

ResourceList




Copy data into cache

Copy file

52625-EMD-002, Rev. 02


AMASS

DiskIndexFile

StagingDataList




EcDsStFtpServer


EcDsStmgtGui


FTP Daemon

ResourceList




Release lien; manage disk information

53625-EMD-002, Rev. 02


AMASS

DiskIndexFile

StagingDataList




EcDsStFtpServer


EcDsStmgtGui


FTP Daemon

ResourceList




Update

cache

information

54625-EMD-002, Rev. 02


AMASS

DiskIndexFile

StagingDataList




EcDsStFtpServer


EcDsStmgtGui


FTP Daemon

ResourceList





55625-EMD-002, Rev. 02


AMASS

DiskIndexFile

StagingDataList




EcDsStFtpServer


EcDsStmgtGui


FTP Daemon

ResourceList



DistributionTransferFunction


56625-EMD-002, Rev. 02


AMASS

DiskIndexFile

StagingDataList




EcDsStFtpServer


EcDsStmgtGui


FTP Daemon

ResourceList




Request capacity; FTP pull/push;

create_allocate/attach; claim

ownership; destroy_deallocate;

link file

57625-EMD-002, Rev. 02


AMASS

DiskIndexFile

StagingDataList




EcDsStFtpServer


EcDsStmgtGui


FTP Daemon

ResourceList




Create_allocate/attach;

destroy_deallocate; link file

58625-EMD-002, Rev. 02


AMASS

DiskIndexFile

StagingDataList




EcDsStFtpServer


EcDsStmgtGui


FTP Daemon

ResourceList




Capacity

Attach to staging disk;

claim ownership; request capacity

59625-EMD-002, Rev. 02


AMASS

DiskIndexFile

StagingDataList




EcDsStFtpServer


EcDsStmgtGui


FTP Daemon

ResourceList




FTP pull; FTP push

60625-EMD-002, Rev. 02


AMASS

DiskIndexFile

StagingDataList




EcDsStFtpServer


EcDsStmgtGui


FTP Daemon

ResourceList




MkDir; FTP into cache; link file

61625-EMD-002, Rev. 02


AMASS

DiskIndexFile

StagingDataList




EcDsStFtpServer


EcDsStmgtGui


FTP Daemon

ResourceList




FTP push

directly to user

62625-EMD-002, Rev. 02


AMASS

DiskIndexFile

StagingDataList




EcDsStFtpServer


EcDsStmgtGui


FTP Daemon

ResourceList




Attach to staging disk; detach from staging disk

63625-EMD-002, Rev. 02


AMASS

DiskIndexFile

StagingDataList




EcDsStFtpServer


EcDsStmgtGui


FTP Daemon

ResourceList




Modify information for disk/pull area

64625-EMD-002, Rev. 02


AMASS

DiskIndexFile

StagingDataList




EcDsStFtpServer


EcDsStmgtGui


FTP Daemon

ResourceList





65625-EMD-002, Rev. 02


• Data Distribution (DDIST) CSCI– Monitors and controls processing of requests for internal and

external electronic distributions; distributions on physical media (8mm tape, CD-ROM, DVD, Digital Linear Tape) are handled as Product Distribution System (PDS) requests via FTPPush onto a PDS working directory, from which PDS reads the data for copy to hard media

– Sends e-mail notifications– Supports distribution of externally subsetted products– Five major components

- Data Distribution Server - provides control and coordination for data distribution through request processing

- Data Distribution GUI - allows operations staff to initiate, track, and manipulate distribution requests

- Data Base - contains the request list; updates and provides the request configuration

DDIST

Collection

66625-EMD-002, Rev. 02


• Data Distribution (DDIST) CSCI– Five major components (Cont.)

- External Product Dispatcher (EPD) - receives products from external subsetter and transfers them via DCLI to DDIST

- DDIST Command Line Interface (DCLI) - submits distribution requests for distribution of externally subsetted products

67625-EMD-002, Rev. 02

EcDsDdistGui

ASE (Sybase)

STMGT/DDISTDatabase

Ingest Local Disk

EXTERNAL

SUBSETTE

R

EcDsDdDCLISubsetterLocation(e.g., DAAC)

EPD Server(epdserver)

EcDsDistributionServer

Subsystems and CSCIs: DSS DDIST Architecture and Interfaces

68625-EMD-002, Rev. 02

EcDsDdistGui

ASE (Sybase)

STMGT/DDISTDatabase

Ingest Local Disk

EXTERNAL

SUBSETTE

R





Distribution request; push data instructions; Acquire

subsetted data

69625-EMD-002, Rev. 02

EcDsDdistGui

ASE (Sybase)

STMGT/DDISTDatabase

Ingest Local Disk

EXTERNAL

SUBSETTE

R





Create_Att; Create_All; Destroy_Dea; Link files; Retrieve data; Claim staging area; Get capacity (of a media type); FTP Push/Pull

70625-EMD-002, Rev. 02

EcDsDdistGui

ASE (Sybase)

STMGT/DDISTDatabase

Ingest Local Disk

EXTERNAL

SUBSETTE

R





Product Delivery Record (via FTPpull);

subsetting output data

Return status (PAN or PDRD) *

*PAN Product Acceptance NotificationPDRD Product Delivery Record Discrepancy

71625-EMD-002, Rev. 02

EcDsDdistGui

ASE (Sybase)

STMGT/DDISTDatabase

Ingest Local Disk

EXTERNAL

SUBSETTE

R





Distribution Request

72625-EMD-002, Rev. 02

EcDsDdistGui

ASE (Sybase)

STMGT/DDISTDatabase

Ingest Local Disk

EXTERNAL

SUBSETTE

R





Request management

services (e.g., user

profile, order/request

information)

User profile

information;

order/request

information

73625-EMD-002, Rev. 02

EcDsDdistGui

ASE (Sybase)

STMGT/DDISTDatabase

Ingest Local Disk

EXTERNAL

SUBSETTE

R





Request communication services; Distribution Notification/Information (via Email to user, external subsetter, or EcInEmailGWServer)

Configuration parameters; server location information

74625-EMD-002, Rev. 02

EcDsDdistGui

ASE (Sybase)

STMGT/DDISTDatabase

Ingest Local Disk

EXTERNAL

SUBSETTE

R





Push Data (from another mode) to Ingest Local Disk (via staging in STMGT and then FTP service)

75625-EMD-002, Rev. 02

EcDsDdistGui

ASE (Sybase)

STMGT/DDISTDatabase

Ingest Local Disk

EXTERNAL

SUBSETTE

R





Status and error

information

Cancel, Resume, or Suspend request; Set Priority; Configuration control information (e.g.,distribution polling rate)

76625-EMD-002, Rev. 02

EcDsDdistGui

ASE (Sybase)

STMGT/DDISTDatabase

Ingest Local Disk

EXTERNAL

SUBSETTE

R





Set Data Distribution parameters (queues, thresholds, suspend, and resume)

77625-EMD-002, Rev. 02

EcDsDdistGui

ASE (Sybase)

STMGT/DDISTDatabase

Ingest Local Disk

EXTERNAL

SUBSETTE

R





Data Distribution

parameters (queues,

thresholds, suspend

and resume status)

78625-EMD-002, Rev. 02


DataData Pool

(DPL)DATAPOOL

ElectronicProducts


(DPS) Data Proc

0110001011001011001011010111010110001011

InventorySearch


Archive


SDSRVDDIST

STMGT

Inven-toryDB

Collection

Insertand

Retrieve


PUSH


PlanningSubsystem

(PLS) Planning

abc

def

ghi

jkl

mno

PULLSystem Mgmt

SubsystemMSS

AcquireRequests

ProductOrders



Meta-data

ProductRequests

EMOS


Searchand

Access


Data MgmtSubsystem

(DMS)


Searchand

Download





Data

DataIngest




SDSRVDDIST

STMGT

Inven-toryDB

Collection

Storage Management(STMGT) CSCI

Cache/StagingManager ProcessEcDsStCacheManagerServer


MediaServer Process

EcDsSt8MMServerEcDsStDTFServerEcDsStFtpServer

EcDsStCopyServer

Pull MonitorProcess


Archive ServerEcDsStArchiveServer

EcDsStmgtGui

Data BaseSybase Server (COTS)

Request ManagerEcDsStRequest Manager Server

Data Distribution(DDIST) CSCI

DataDistribution


DataDistribution GUI

EcDsDdistGui


EPD Serverepdserver DCLI

EcDsDdDCLI

Science Data Server(SDSRV) CSCI

ScienceData Server


HDF EOS ServerEcDsHdfEosServer

Science Data Server GUIEcDsSdsrvGui

SybaseSybase Server (COTS)

Granule DeletionTool

EcDsBulkDelete.pl

SQSSpatial Query Server (COTS)

SDSRV Command Line Interface

EcDsSCLI

79625-EMD-002, Rev. 02


DataData Pool

(DPL)DATAPOOL

ElectronicProducts


(DPS) Data Proc

0110001011001011001011010111010110001011

InventorySearch


Archive


SDSRVDDIST

STMGT

Inven-toryDB

Collection

Insertand

Retrieve


PUSH


PlanningSubsystem

(PLS) Planning

abc

def

ghi

jkl

mno

PULLSystem Mgmt

SubsystemMSS

AcquireRequests

ProductOrders



Meta-data

ProductRequests

EMOS


Searchand

Access


Data MgmtSubsystem

(DMS)


Searchand

Download





Data

DataIngest


Subsystems and CSCIs: INS

IngestSubsystem

(INS) Ingest

80625-EMD-002, Rev. 02

Subsystems and CSCIs: INS

• Ingest Subsystem (INS)– Transfer of data into ECS (SDPS repositories) in accordance

with approved ICDs– Supports varied data formats and structures– Ingest Client: A set of ingest software configured for

requirements of a specific situation– Ingest clients perform data preprocessing, such as format

conversion, metadata extraction, and metadata validation on incoming data

– Data staged to one of two areas- Level 0 (L0) data from ongoing missions, and EDOS ancillary data,

staged to INS working storage area- Non-L0 data (e.g., non-EDOS ancillary data, L1A-L4 data) staged

directly to DSS working storage area – Uses several COTS tools: RogueWave class libraries, Sybase

relational database, CCS Middleware Client

IngestSubsystem Ingest

81625-EMD-002, Rev. 02

Subsystems and CSCIs: INS (Cont.)

• Ingest (INGST) CSCI– Gets data by various methods and transfers the data into ECS

- Polling: transfer of data from predetermined network locations which Ingest periodically checks for new data- With Delivery Record- Without Delivery Record

- Media: reading data from physical media; uses GUI- Cross-Mode Ingest: E-mail distribution notification used to create

a Delivery Record File for Polling with Delivery Record– Stores and manages request information– Provides for data preprocessing and insertion

82625-EMD-002, Rev. 02


• Ingest (INGST) CSCI (Cont.)– Six major components

- Polling Ingest Client Interface - creates polling request, detects new files in a specified external location, creates and submits ingest request

- Media Ingest Interface - provides operators ability to perform ingest from physical media

- Cross-Mode Ingest Interface - provides an E-mail gateway server to receive E-mail distribution notifications and store them as files in a location for polling with delivery record

- Ingest Request Manager - manages ingest request traffic and processing

- Ingest Granule Server - provides services for required preprocessing of data and subsequent insertion into Data Server

- Ingest Database - stores and provides access to Ingest Subsystem internal data (e.g., Request Status, History Logs)

83625-EMD-002, Rev. 02

INSINGST

EcInGUI

ASE(Sybase)

ExternalDataProviders

EcInPolling

EcInReqMgr

IngestDatabase

DAAC IngestTechnician

EcInEmailGWServer

EcInTapeReaderGUI

EcInGran

LocalDisk

Subsystems and CSCIs: INS (Cont.)Architecture and Interfaces

84625-EMD-002, Rev. 02

INSINGST

EcInGUI

ASE(Sybase)


EcInPolling

EcInReqMgr

IngestDatabase


EcInEmailGWServer

EcInTapeReaderGUI

EcInGran

LocalDisk


Store datato beIngested

85625-EMD-002, Rev. 02

INSINGST

EcInGUI

ASE(Sybase)


EcInPolling

EcInReqMgr

IngestDatabase


EcInEmailGWServer

EcInTapeReaderGUI

EcInGran

LocalDisk


Detect and

read data

86625-EMD-002, Rev. 02

INSINGST

EcInGUI

ASE(Sybase)


EcInPolling

EcInReqMgr

IngestDatabase


EcInEmailGWServer

EcInTapeReaderGUI

EcInGran

LocalDisk


Send request

87625-EMD-002, Rev. 02

INSINGST

EcInGUI

ASE(Sybase)


EcInPolling

EcInReqMgr

IngestDatabase


EcInEmailGWServer

EcInTapeReaderGUI

EcInGran

LocalDisk


*PAN Product Acceptance NotificationPDRD Product Delivery Record Discrepancy

Return status

(PAN or PDRD) *

88625-EMD-002, Rev. 02

INSINGST

EcInGUI

ASE(Sybase)


EcInPolling

EcInReqMgr

IngestDatabase


EcInEmailGWServer

EcInTapeReaderGUI

EcInGran

LocalDisk


Clean up directory (Ingest local disk only)

89625-EMD-002, Rev. 02

INSINGST

EcInGUI

ASE(Sybase)


EcInPolling

EcInReqMgr

IngestDatabase


EcInEmailGWServer

EcInTapeReaderGUI

EcInGran

LocalDisk


Monitor and control Ingest functions

90625-EMD-002, Rev. 02

INSINGST

EcInGUI

ASE(Sybase)


EcInPolling

EcInReqMgr

IngestDatabase


EcInEmailGWServer

EcInTapeReaderGUI

EcInGran

LocalDisk


Monitor and

control Tape

Reader functions

91625-EMD-002, Rev. 02

INSINGST

EcInGUI

ASE(Sybase)


EcInPolling

EcInReqMgr

IngestDatabase


EcInEmailGWServer

EcInTapeReaderGUI

EcInGran

LocalDisk


Control

peripherals

Create staging disk;control peripherals

92625-EMD-002, Rev. 02

INSINGST

EcInGUI

ASE(Sybase)


EcInPolling

EcInReqMgr

IngestDatabase


EcInEmailGWServer

EcInTapeReaderGUI

EcInGran

LocalDisk


Store datato beIngested

93625-EMD-002, Rev. 02

INSINGST

EcInGUI

ASE(Sybase)


EcInPolling

EcInReqMgr

IngestDatabase


EcInEmailGWServer

EcInTapeReaderGUI

EcInGran

LocalDisk


Granule processing/cancellation requests

Completion notification

94625-EMD-002, Rev. 02

INSINGST

EcInGUI

ASE(Sybase)


EcInPolling

EcInReqMgr

IngestDatabase


EcInEmailGWServer

EcInTapeReaderGUI

EcInGran

LocalDisk


Allocate and transfer files to Staging Disk; Get MCF; Validate metadata; Data granule insert; Data search

95625-EMD-002, Rev. 02

INSINGST

EcInGUI

ASE(Sybase)


EcInPolling

EcInReqMgr

IngestDatabase


EcInEmailGWServer

EcInTapeReaderGUI

EcInGran

LocalDisk


Update Ingest

tunable parameters; Data for

reports

Record and retrieve request information

96625-EMD-002, Rev. 02

INSINGST

EcInGUI

ASE(Sybase)


EcInPolling

EcInReqMgr

IngestDatabase


EcInEmailGWServer

EcInTapeReaderGUI

EcInGran

LocalDisk


Update file

and granule

information

97625-EMD-002, Rev. 02

INSINGST

EcInGUI

ASE(Sybase)


EcInPolling

EcInReqMgr

IngestDatabase


EcInEmailGWServer

EcInTapeReaderGUI

EcInGran

LocalDisk


Query and obtain data

98625-EMD-002, Rev. 02

INSINGST

EcInGUI

ASE(Sybase)


EcInPolling

EcInReqMgr

IngestDatabase


EcInEmailGWServer

EcInTapeReaderGUI

EcInGran

LocalDisk


Push data (fromanothermode)

99625-EMD-002, Rev. 02

INSINGST

EcInGUI

ASE(Sybase)


EcInPolling

EcInReqMgr

IngestDatabase


EcInEmailGWServer

EcInTapeReaderGUI

EcInGran

LocalDisk


Send Distribution

Notification

100625-EMD-002, Rev. 02

INSINGST

EcInGUI

ASE(Sybase)


EcInPolling

EcInReqMgr

IngestDatabase


EcInEmailGWServer

EcInTapeReaderGUI

EcInGran

LocalDisk


Detect data;transfer PDR

101625-EMD-002, Rev. 02

INSINGST

EcInGUI

ASE(Sybase)


EcInPolling

EcInReqMgr

IngestDatabase


EcInEmailGWServer

EcInTapeReaderGUI

EcInGran

LocalDisk


Request

Communications

Support

102625-EMD-002, Rev. 02


DataData Pool

(DPL)DATAPOOL

ElectronicProducts


(DPS) Data Proc

0110001011001011001011010111010110001011

InventorySearch


Archive


SDSRVDDIST

STMGT

Inven-toryDB

Collection

Insertand

Retrieve


PUSH


PlanningSubsystem

(PLS) Planning

abc

def

ghi

jkl

mno

PULLSystem Mgmt

SubsystemMSS

AcquireRequests

ProductOrders



Meta-data

ProductRequests

EMOS


Searchand

Access


Data MgmtSubsystem

(DMS)


Searchand

Download





Data

DataIngest



IngestSubsystem

(INS) IngestIngest(INGST) CSCI

Polling IngestClient Interface

ProcessEcInPolling

MediaIngest Interface

ProcessEcInGUI

EcInTapeReaderGUI

IngestRequest Manager

ProcessEcInReqMgr

IngestGranule Server

ProcessEcInGran

IngestDatabase

Sybase Server (COTS)

Cross-ModeIngest Interface

EcInEmailGWServer

103625-EMD-002, Rev. 02


DataData Pool

(DPL)DATAPOOL

ElectronicProducts


(DPS) Data Proc

0110001011001011001011010111010110001011

InventorySearch


Archive


SDSRVDDIST

STMGT

Inven-toryDB

Collection

Insertand

Retrieve


PUSH


PlanningSubsystem

(PLS) Planning

abc

def

ghi

jkl

mno

PULLSystem Mgmt

SubsystemMSS

AcquireRequests

ProductOrders



Meta-data

ProductRequests

EMOS


Searchand

Access


Data MgmtSubsystem

(DMS)


Searchand

Download





Data

DataIngest


Subsystems and CSCIs: SSS


(SSS)

104625-EMD-002, Rev. 02

Subsystems and CSCIs: SSS

• Spatial Subscription Server (SSS)– Creating, viewing, updating Subscriptions (specification of an

action and an event that initiates the action)- Actions: Notification, Distribution, Data Pool Insert- Events: Granule Insert, Granule Deletion, Metadata Update

– Creating, viewing, deleting Bundling Orders (specification of distribution packages and criteria for package completion)- Minimum bundle size- Minimum granule count- Maximum bundle age- Bundling order information stored in Order Manager database

– Subscription processing triggered by appearance of events in Science Data Server database- Identify all subscriptions to the specified event- Process the actions defined in the subscriptions


(SSS)

105625-EMD-002, Rev. 02

Subsystems and CSCIs: SSS (Cont.)

• Spatial Subscription Server (SSS) (Cont.)– Uses several COTS tools: Netscape Navigator, Sun ONE Web

Server, Sybase ASE• Spatial Subscription Server (NBSRV) CSCI (a.k.a. NSBRV

CSCI)– Provides a Graphical User Interface (GUI) and a set of drivers

for implementing subscription functions– Six major components

- Spatial Subscription Server database - repository for all data created expressly for use by the NBSRV

- Subscription GUI - tool for entering, modifying, or deleting subscriptions and bundling orders

- Event Queue Monitor - multiple instances of a Perl script that monitors the event queue for new arrivals

- Action Queue Monitor - multiple instances of a Perl script that monitors the action queue for new arrivals

106625-EMD-002, Rev. 02


• Spatial Subscription Server (NBSRV) CSCI (a.k.a. NSBRV CSCI) (Cont.)– Six major components (Cont.)

- Recovery Driver - Monitors logs for stalled events or actions; re-enqueues stalled events/actions

- Deletion Driver - Works off a deletion queue to purge the database of outdated information (e.g., completed events/actions)

107625-EMD-002, Rev. 02

Subsystems and CSCIs: SSS (Cont.) NBSRV Architecture and Interfaces

ASE(Sybase)

SSSDatabase

SSSNBSRV

EcNbSubscribedEventDriver.pl

WebBrowser EcNbGUISun ONE

WebServer

EcNbDeleteRequestDriver.pl

EcNbRecoverDriver.plEcNbActionDriver.pl

DAACOperator

108625-EMD-002, Rev. 02


ASE(Sybase)

SSSDatabase

SSSNBSRV



WebServer



DAACOperator

List/view subscribable events, subscriptions, actions, statistics, bundling orders

Create/modify/delete subscriptions

for Data Pool insert o

r distribution/

notification; create/modify/delete

bundling orders

109625-EMD-002, Rev. 02


ASE(Sybase)

SSSDatabase

SSSNBSRV



WebServer



DAACOperator

Create (bundling) order

Order ID (for bundling order)

110625-EMD-002, Rev. 02

ASE(Sybase)

SSSDatabase

SSSNBSRV



WebServer



DAACOperator


Retrieve data

Update data

111625-EMD-002, Rev. 02

ASE(Sybase)

SSSDatabase

SSSNBSRV



WebServer



DAACOperator


Bundling order information

112625-EMD-002, Rev. 02

ASE(Sybase)

SSSDatabase

SSSNBSRV



WebServer



DAACOperator


Event for subscribed event queue in SSS Database (based on activation of trigger upon insert of qualified granule)

113625-EMD-002, Rev. 02

ASE(Sybase)

SSSDatabase

SSSNBSRV



WebServer



DAACOperator


Monitor event queue and

detect event;

Get metadata for granule that triggered event

metadata

(from SDSRV database);

identify subscriptions for

which a granule qualifies

114625-EMD-002, Rev. 02

ASE(Sybase)

SSSDatabase

SSSNBSRV



WebServer



DAACOperator


Data Pool Insert action; browse granule (direct write to Data Pool database)

115625-EMD-002, Rev. 02

ASE(Sybase)

SSSDatabase

SSSNBSRV



WebServer



DAACOperator


Monitor action queue and

detect event; get action(s)

for subscription

116625-EMD-002, Rev. 02

ASE(Sybase)

SSSDatabase

SSSNBSRV



WebServer



DAACOperator


Create (non-bundling) order

Order ID and request ID (for non-bundling order)Note: The request ID is used for an immediate distribution action.

117625-EMD-002, Rev. 02

ASE(Sybase)

SSSDatabase

SSSNBSRV



WebServer



DAACOperator


Acquire request

118625-EMD-002, Rev. 02

ASE(Sybase)

SSSDatabase

SSSNBSRV



WebServer



DAACOperator


Email text file

119625-EMD-002, Rev. 02

ASE(Sybase)

SSSDatabase

SSSNBSRV



WebServer



DAACOperator


Read deletion queue; clean

up database tables to purge

outdated information (e.g.,

completed events/actions)

120625-EMD-002, Rev. 02

ASE(Sybase)

SSSDatabase

SSSNBSRV



WebServer



DAACOperator


Monitor logs for

stalled processing

of events/actions;

re-enqueue stalled

events/actions

121625-EMD-002, Rev. 02


DataData Pool

(DPL)DATAPOOL

ElectronicProducts


(DPS) Data Proc

0110001011001011001011010111010110001011

InventorySearch


Archive


SDSRVDDIST

STMGT

Inven-toryDB

Collection

Insertand

Retrieve


PUSH


PlanningSubsystem

(PLS) Planning

abc

def

ghi

jkl

mno

PULLSystem Mgmt

SubsystemMSS

AcquireRequests

ProductOrders



Meta-data

ProductRequests

EMOS


Searchand

Access


Data MgmtSubsystem

(DMS)


Searchand

Download





Data

DataIngest




(SSS)

Spatial Subscription Server(NBSRV) CSCI

Action DriverEcNbActionDriver.pl

NBSRV GUIEcNbGUI

Subscribed EventDriver


Deletion DriverEcNbDeleteRequestDriver.pl

SSS (NBSRV)Database


Recovery DriverEcNbRecoverDriver.pl

122625-EMD-002, Rev. 02


DataData Pool

(DPL)DATAPOOL

ElectronicProducts


(DPS) Data Proc

0110001011001011001011010111010110001011

InventorySearch


Archive


SDSRVDDIST

STMGT

Inven-toryDB

Collection

Insertand

Retrieve


PUSH


PlanningSubsystem

(PLS) Planning

abc

def

ghi

jkl

mno

PULLSystem Mgmt

SubsystemMSS

AcquireRequests

ProductOrders



Meta-data

ProductRequests

EMOS


Searchand

Access


Data MgmtSubsystem

(DMS)


Searchand

Download





Data

DataIngest


Subsystems and CSCIs: DPL

Data Pool(DPL)

DATAPOOL

123625-EMD-002, Rev. 02

Subsystems and CSCIs: DPL

• Data Pool (DPL)– An on-line repository of selected granules with associated

metadata and, if available, browse granules– Accessible through a web browser– Accessible through FTP– Data downloadable via FTP– Provides easy-to-use drill-down web user interface– Included in EDG data search results– Populated by subscriptions for Data Pool insert– Stages data for OMS FTP distributions– Uses several COTS tools: Netscape Navigator, Sun ONE Web

Server, Apache Web Server, wuftp (COTS FTP Server), Sybase ASE

Data Pool(DPL)

DATAPOOL

124625-EMD-002, Rev. 02

Subsystems and CSCIs: DPL (Cont.)

• Data Pool Management (DPMGT) CSCI– Provides on-line cache for access to selected ECS data,

metadata, and browse granules– Permits user search and FTP download through a web interface– Permits user browsing and download during an FTP session– Provides an Insert Utility for insert of data and metadata– Serves as staging cache for OMS FTP distributions (push and

pull)– Manages archive drive resources (with OMS help)

125625-EMD-002, Rev. 02


• Data Pool Management (DPMGT) CSCI (Cont.)– Four major components

- Data Pool Insert Utility - consists of four subcomponents- Action Driver schedules insert actions- Insert Utility requests copy to the Data Pool and updates the

inventory- Quick Server, a C++ executable, performs the copy from AMASS- EcDlM2XT, a java executable, translates granule metadata into

XML format- Data Pool Web Access - a web-based GUI that provides easy drill-

down search and FTP download for the user- wuftp - COTS FTP server for user-interactive FTP sessions- Data Pool Database - Sybase database stores Data Pool inventory

and configuration information

126625-EMD-002, Rev. 02

ASE(Sybase)

DPL Database

DPLDPMGT

Data Pool Insert(EcDlInsertUtility,EcDlActionDriver,EcDlQuickServer,EcDlM2XTApp)

DPMNT

wuftp(COTS FTP Server)

Data PoolData PoolDisksDisks

ApacheWeb Server

WebBrowser

FTP Client

Firewall

EcDlWebAccess

Science User

Subsystems and CSCIs: DPL (Cont.)DPMGT Architecture and Interfaces

127625-EMD-002, Rev. 02

ASE(Sybase)

DPL Database

DPLDPMGT


DPMNT



ApacheWeb Server

WebBrowser

FTP Client

Firewall

EcDlWebAccess

Science User


Place an insert on the

Data Pool Insert Action

Queue

Submit granule staging request

128625-EMD-002, Rev. 02

ASE(Sybase)

DPL Database

DPLDPMGT


DPMNT



ApacheWeb Server

WebBrowser

FTP Client

Firewall

EcDlWebAccess

Science User


Query/ update database tables

129625-EMD-002, Rev. 02

ASE(Sybase)

DPL Database

DPLDPMGT


DPMNT



ApacheWeb Server

WebBrowser

FTP Client

Firewall

EcDlWebAccess

Science User


Copy granule,

insert browse;

convert and

store metadata

Request data

insert; request

browse insert;

request metadata

130625-EMD-002, Rev. 02

ASE(Sybase)

DPL Database

DPLDPMGT


DPMNT



ApacheWeb Server

WebBrowser

FTP Client

Firewall

EcDlWebAccess

Science User


Update Data Pool

Inventory

131625-EMD-002, Rev. 02

ASE(Sybase)

DPL Database

DPLDPMGT


DPMNT



ApacheWeb Server

WebBrowser

FTP Client

Firewall

EcDlWebAccess

Science User


User search of

Data Pool to locate

file(s) of interest

132625-EMD-002, Rev. 02

ASE(Sybase)

DPL Database

DPLDPMGT


DPMNT



ApacheWeb Server

WebBrowser

FTP Client

Firewall

EcDlWebAccess

Science User


nfs read for

FTP download

133625-EMD-002, Rev. 02

ASE(Sybase)

DPL Database

DPLDPMGT


DPMNT



ApacheWeb Server

WebBrowser

FTP Client

Firewall

EcDlWebAccess

Science User


Submit hard

media order

134625-EMD-002, Rev. 02

ASE(Sybase)

DPL Database

DPLDPMGT


DPMNT



ApacheWeb Server

WebBrowser

FTP Client

Firewall

EcDlWebAccess

Science User


User ftp

search and

download of

Data Pool

file(s) of

interest

135625-EMD-002, Rev. 02

ASE(Sybase)

DPL Database

DPLDPMGT


DPMNT



ApacheWeb Server

WebBrowser

FTP Client

Firewall

EcDlWebAccess

Science User


FTP search

and download

using web

browser

136625-EMD-002, Rev. 02

ASE(Sybase)

DPL Database

DPLDPMGT


DPMNT



ApacheWeb Server

WebBrowser

FTP Client

Firewall

EcDlWebAccess

Science User


Request metadata

URLs, science data

URLs, browse

URLs, and

expiration dates

Return metadata

URLs, science data

URLs, browse

URLs, and

expiration dates

137625-EMD-002, Rev. 02

ASE(Sybase)

DPL Database

DPLDPMGT


DPMNT



ApacheWeb Server

WebBrowser

FTP Client

Firewall

EcDlWebAccess

Science User


Copy data

138625-EMD-002, Rev. 02

ASE(Sybase)

DPL Database

DPLDPMGT


DPMNT



ApacheWeb Server

WebBrowser

FTP Client

Firewall

EcDlWebAccess

Science User


Data Pool Maintenance and Monitoring

139625-EMD-002, Rev. 02


• Data Pool Maintenance (DPMNT) CSCI– Provides a maintenance GUI that allows operators to monitor

and control Data Pool insert activity and control the Data Pool configuration

– Provides utilities and scripts for Data Pool maintenance– Nine major components

- Data Pool Maintenance (DPM) GUI - a Perl web-based GUI for Data Pool monitoring and control

- Update Granule Expiration Utility - a Perl utility that allows updating the expiration date and retention priority for granules in the Data Pool

- Data Pool Cleanup - a Perl utility that removes expired granules from the Data Pool and database (normally run as a cron job)

- Data Pool Access Statistics Utility (DPASU) - Perl utilities that extract access statistics from logs and roll up access information for storage in the Data Pool database

140625-EMD-002, Rev. 02


• Data Pool Maintenance (DPMNT) CSCI (Cont.)– Nine major components (Cont.)

- Data Pool Archive/Delete/Restore Access Statistics - shell scripts to manage access statistics

- Data Pool Most Recent Insert Utility - a Perl utility that creates files containing information concerning granules recently inserted into the Data Pool

- Data Pool Collection Remapping Utility - a Perl utility that allows DAAC operations staff to remap all data in a Data Pool collection directory from one higher-level collection group directory to another

- Data Pool Density Map Utility - a Perl utility that calculates spatial density map information about Data Pool collections and stores the information in the Data Pool database

- Data Pool Statistics Table Population Utility - a Perl utility that populates Data Pool database tables which maintain counts of granules by collection and collection group for use by the Web Access drill down GUI

141625-EMD-002, Rev. 02DPL

DPMNT

DPMGT

WebBrowser

DAAC Operator

Data Pool Maintenance GUI

EcDlCleanupDataPool.pl

EcDlUpdateGranule.pl

EcDlRollupWebLogs.plEcDlRollupFwFtpLogs.pl

EcDlRemap.pl

Sun ONE Web Server

EcDlDensityMapUtility.pl

EcDlMostRecentInsert.pl

File of RecentGranules

EcDlPopulateStatTables.pl

DlDbArchiveAccessStatDlDbDeleteAccessStatDlDbRestoreAccessStat

Subsystems and CSCIs: DPL (Cont.)DPMNT Architecture and Interfaces

142625-EMD-002, Rev. 02DPL

DPMNT

DPMGT

WebBrowser

DAAC Operator





EcDlRemap.pl

Sun ONE Web Server







Control Data Pool processes

(e.g., Suspend/Resume/

Cancel Data Pool Insert

actions, configure defaults,

add/modify collections and

collection groups)Query/

update

database

tables

View/monitor Data Pool processes

143625-EMD-002, Rev. 02DPL

DPMNT

DPMGT

WebBrowser

DAAC Operator





EcDlRemap.pl

Sun ONE Web Server







Extend retention/modify priority for science data in the Data Pool

144625-EMD-002, Rev. 02DPL

DPMNT

DPMGT

WebBrowser

DAAC Operator





EcDlRemap.pl

Sun ONE Web Server







Invoke cleanup manually (normally it

runs as a cron job)Query database to qualify granules for deletion; delete

145625-EMD-002, Rev. 02DPL

DPMNT

DPMGT

WebBrowser

DAAC Operator





EcDlRemap.pl

Sun ONE Web Server







Extract Data Pool access statistics from web logs and FTP logs; store in database

146625-EMD-002, Rev. 02DPL

DPMNT

DPMGT

WebBrowser

DAAC Operator





EcDlRemap.pl

Sun ONE Web Server







Archive/delete/restore information in the database tables populated by the Data Pool Access Statistics Utility (Rollup)

147625-EMD-002, Rev. 02DPL

DPMNT

DPMGT

WebBrowser

DAAC Operator





EcDlRemap.pl

Sun ONE Web Server







Remap data from one higher-level collection groupto another

148625-EMD-002, Rev. 02DPL

DPMNT

DPMGT

WebBrowser

DAAC Operator





EcDlRemap.pl

Sun ONE Web Server







Create files containing information about recently inserted granules

149625-EMD-002, Rev. 02DPL

DPMNT

DPMGT

WebBrowser

DAAC Operator





EcDlRemap.pl

Sun ONE Web Server







Calculate spatial

density map

information about

collections; store

in the database

150625-EMD-002, Rev. 02DPL

DPMNT

DPMGT

WebBrowser

DAAC Operator





EcDlRemap.pl

Sun ONE Web Server







Populate database

tables that

maintain counts of

granules by

collection/group

151625-EMD-002, Rev. 02


DataData Pool

(DPL)DATAPOOL

ElectronicProducts


(DPS) Data Proc

0110001011001011001011010111010110001011

InventorySearch


Archive


SDSRVDDIST

STMGT

Inven-toryDB

Collection

Insertand

Retrieve


PUSH


PlanningSubsystem

(PLS) Planning

abc

def

ghi

jkl

mno

PULLSystem Mgmt

SubsystemMSS

AcquireRequests

ProductOrders



Meta-data

ProductRequests

EMOS


Searchand

Access


Data MgmtSubsystem

(DMS)


Searchand

Download





Data

DataIngest



Data Pool(DPL)

DATAPOOL

Data Pool Management(DPMGT) CSCI

COTS FTPServer

wuftp

Data PoolWeb Access GUI

EcDlWebAccess

Data PoolDatabase


Data PoolInsert Utility

EcDlInsertUtilityEcDlActionDriverEcDlQuickServer

EcDlM2XTApp

Data Pool Maintenance(DPMNT) CSCI

Data PoolUpdate Granule

ExpirationEcDlUpdateGranule.pl

DPM GUIEcDlDpmHomepage.pl

Data Pool AccessStatistics UtilityEcDlRollupWebLogs.plEcDlRollupFtpLogs.pl

Data PoolArchive Access

Statistics ScriptsDlDbArchiveAccessStatDlDbDeleteAccessStat

DlDbRestoreAccessStat

Data Pool CleanupEcDlCleanupDataPool.pl

Data PoolMost RecentInsert Utility


Data PoolCollection

Remapping UtilityEcDlRemap.pl

Data PoolDensity Map

UtilityEcDlDensityMapUtility.pl

Data PoolStatistics Table

Population UtilityEcDlPopulateStatTables.pl

152625-EMD-002, Rev. 02


DataData Pool

(DPL)DATAPOOL

ElectronicProducts


(DPS) Data Proc

0110001011001011001011010111010110001011

InventorySearch


Archive


SDSRVDDIST

STMGT

Inven-toryDB

Collection

Insertand

Retrieve


PUSH


PlanningSubsystem

(PLS) Planning

abc

def

ghi

jkl

mno

PULLSystem Mgmt

SubsystemMSS

AcquireRequests

ProductOrders



Meta-data

ProductRequests

EMOS


Searchand

Access


Data MgmtSubsystem

(DMS)


Searchand

Download





Data

DataIngest


Subsystems and CSCIs: CLS


153625-EMD-002, Rev. 02

Subsystems and CSCIs: CLS

• Client Subsystem (CLS)– User access to ECS services for ASTER

- Permits Data Acquisition Request to task ASTER instrument- Supports request of ASTER On-demand Products

– Provides user authentication and User Profile information to the Search and Order tool- Search and retrieval of data are performed by the EOS Data

Gateway (Version 0 Web Client)– Includes applications programs accessible via user interfaces

- EOSView- ASTER Data Acquisition Request (DAR) Tool- On-Demand Form Request Manager (ODFRM)

– Uses several COTS tools: Netscape Navigator, Netscape Enterprise Server, XVT (widget set and development tool for EOSView), and Interactive Data Language (IDL) (used in EOSView visualization features)

Client

154625-EMD-002, Rev. 02

Subsystems and CSCIs: CLS (Cont.)

• Workbench (WKBCH) CSCI– Includes a set of application programs that implement

functions of the CLS science user interface– Release 7 Workbench includes two tools

- EOSView (X/Motif-based)- ASTER DAR Tool (Java/HTML-based)

155625-EMD-002, Rev. 02

Science/DAAC User

CLS

WKBCH

EOSView

EcClWbJdt

HDFFile

WebBrowser

ODFRM CSCI

Subsystems and CSCIs: CLS (Cont.)WKBCH Architecture and Interfaces

156625-EMD-002, Rev. 02

Science/DAAC User

CLS

WKBCH

EOSView

EcClWbJdt

HDFFile

WebBrowser

ODFRM CSCI


Read/Display

HDF File

Enter

HDF File

Name

157625-EMD-002, Rev. 02

Science/DAAC User

CLS

WKBCH

EOSView

EcClWbJdt

HDFFile

WebBrowser

ODFRM CSCI


Invoke browser; Enter

User ID information;

submit service requests;

enter DAR parameters

158625-EMD-002, Rev. 02

Science/DAAC User

CLS

WKBCH

EOSView

EcClWbJdt

HDFFile

WebBrowser

ODFRM CSCI


Send User ID

information; Submit

service requests,

Send DAR parameters

159625-EMD-002, Rev. 02

Science/DAAC User

CLS

WKBCH

EOSView

EcClWbJdt

HDFFile

WebBrowser

ODFRM CSCI


User Profile Information

Request User Profile

160625-EMD-002, Rev. 02

Science/DAAC User

CLS

WKBCH

EOSView

EcClWbJdt

HDFFile

WebBrowser

ODFRM CSCI


Session ID; DAR ID; Results of ASTER DAR search

Submit/Modify Data Acquisition Request (DAR) or submit request to search ASTER DAR database [to EcGwDARServer]; Subscription Request [to EcSbSubServer]; User Information [through CCS Middleware Support]; allvia EcCsMojoGateway

161625-EMD-002, Rev. 02

Science/DAAC User

CLS

WKBCH

EOSView

EcClWbJdt

HDFFile

WebBrowser

ODFRM CSCI


Submit call to attach aData Processing Request(DPR) to a DAR

162625-EMD-002, Rev. 02


• Desktop (DESKT) CSCI– Provides a gateway server for communication with MSS User

Registration Server to support seamless user registration through the EOS Data Gateway (EDG) web client and to obtain or update user profile information

– User Profile Gateway - provides user profile information to the EDG for ECS users- User authentication- Submit/Update user information in profile

163625-EMD-002, Rev. 02

Subsystems and CSCIs: CLS (Cont.)DESKT Architecture and Interfaces

Science/DAAC User

CLSDESKT

WebBrowser

EcClDtUserProfileGatewayUserProfileFile(profile.odl)

(Designcontrolled underseparate contract)

V0 IMS

SMC Location

164625-EMD-002, Rev. 02


Science/DAAC User

CLSDESKT

WebBrowser



V0 IMS

SMC Location

Send User Registration request; Get/Update User Profile; Send password change; Send Search & Order data parameters

Invoke browser; Submit

User Registration request;

Request/Update User Profile;

Request password change;

Enter Search & Order data

parameters

165625-EMD-002, Rev. 02


Science/DAAC User

CLSDESKT

WebBrowser



V0 IMS

SMC LocationUser Profile

Information

Submit/Update

User Profile/Pass-

word information

166625-EMD-002, Rev. 02


Science/DAAC User

CLSDESKT

WebBrowser



V0 IMS

SMC LocationSearch Inventory; Request Product

Results ofSearch

167625-EMD-002, Rev. 02


• On-Demand Form Request Manager (ODFRM) CSCI– Supports attachment of a Data Processing Request (DPR) to an

ASTER Data Acquisition Request (a required capability that is not used)

– ODFRM HTML pages and Common Gateway Interface (CGI) programs

– Creation of an on-demand processing request and its submission to the Planning Subsystem (PLS) is now done through the EOS Data Gateway (EDG) tool

– ASTER on-demand products- ASTER L1B*- ASTER DEM (Digital Elevation Model)*- ASTER higher-level products (AST_04, AST_05, AST_06V,

AST_06T, AST_06S, AST_07S, AST_07V, AST_09T, AST_09V, AST_09S, AST_08)

* Note: Requires special privilege (in User Profile) to use ODFRM to order this product

168625-EMD-002, Rev. 02

CLS

ODFRM

ODFRM

ECS Users

WebBrowser

Sun ONEWeb Server

EcClOdRequest(CGI Program)

WKBCH CSCI

Subsystems and CSCIs: CLS (Cont.)ODFRM Architecture and Interfaces

169625-EMD-002, Rev. 02

CLS

ODFRM

ODFRM

ECS Users

WebBrowser

Sun ONEWeb Server


WKBCH CSCI


Request user profile info;

create On-Demand

Product (ODP) Requests

Invoke browser; request

user profile info; enter

on-demand products info

Submit on-demand request using an

attached DPR

170625-EMD-002, Rev. 02

CLS

ODFRM

ODFRM

ECS Users

WebBrowser

Sun ONEWeb Server


WKBCH CSCI


Pass validated ODP request; request user profile

Submit on-demand product requests; submit standing on-demand ordersRequest user

profile; request order/request tracking

Request media

options

171625-EMD-002, Rev. 02

CLS

ODFRM

ODFRM

ECS Users

WebBrowser

Sun ONEWeb Server


WKBCH CSCI


Return media options

Return user

profile; return

order status

Display user profile;

display media options;

display order status

172625-EMD-002, Rev. 02


DataData Pool

(DPL)DATAPOOL

ElectronicProducts


(DPS) Data Proc

0110001011001011001011010111010110001011

InventorySearch


Archive


SDSRVDDIST

STMGT

Inven-toryDB

Collection

Insertand

Retrieve


PUSH


PlanningSubsystem

(PLS) Planning

abc

def

ghi

jkl

mno

PULLSystem Mgmt

SubsystemMSS

AcquireRequests

ProductOrders



Meta-data

ProductRequests

EMOS


Searchand

Access


Data MgmtSubsystem

(DMS)


Searchand

Download





Data

DataIngest



Desktop(DESKT) CSCI

User ProfileGateway

EcClDtUserProfileGateway

Workbench(WKBCH) CSCI

HDF ProductViewing Tool

EOSView

ASTER DARTool

EcClWbJdt

On-Demand FormRequest Manager

(ODFRM) CSCI

HTML UserInterface

ODFRM

CGIProgramsEcClOdRequest


173625-EMD-002, Rev. 02


DataData Pool

(DPL)DATAPOOL

ElectronicProducts


(DPS) Data Proc

0110001011001011001011010111010110001011

InventorySearch


Archive


SDSRVDDIST

STMGT

Inven-toryDB

Collection

Insertand

Retrieve


PUSH


PlanningSubsystem

(PLS) Planning

abc

def

ghi

jkl

mno

PULLSystem Mgmt

SubsystemMSS

AcquireRequests

ProductOrders



Meta-data

ProductRequests

EMOS


Searchand

Access


Data MgmtSubsystem

(DMS)


Searchand

Download





Data

DataIngest


Subsystems and CSCIs: DMS

Data MgmtSubsystem

(DMS)

174625-EMD-002, Rev. 02

Subsystems and CSCIs: DMS

• Data Management Subsystem (DMS)– Provides one-way catalog interoperability between ECS and the

V0 Information Management System (IMS)– Supplies gateway processes to translate requests between V0

protocol and ECS– Maintains a Data Dictionary that stores ECS data collection

information (i.e., collection metadata, attributes, valid keywords) and mappings between this information and V0 to permit translation of requests between the systems

– Uses several COTS tools: RogueWave class libraries, Builder Xcessory (GUI builder tool), and Sybase ASE Server (for Data Dictionary database search and update)

Data MgmtSubsystem

175625-EMD-002, Rev. 02

Subsystems and CSCIs: DMS(Cont.)

• Data Dictionary (DDICT) CSCI– Manages definitions of data collections including metadata,

data domains (valid values), and data location– Stored in a relational Database Management System (DBMS)– Three major components

- Data Dictionary Server - provides DDICT client processes the ability to perform data searches, inserts, updates, or deletes to the DDICT database

- Data Dictionary Maintenance Tool - provides a GUI to insert, update, or delete schema information held in the DDICT database,and allows operations staff to modify database attributes (e.g.,valids, mapping)

- Data Dictionary ASE Server - COTS database server

176625-EMD-002, Rev. 02

DMS

DDICT

EcDmDdMaintenanceTool

ASE(Sybase)

EcDmDictServiceDatabase

EcDmDictServer

V0 GTWAY CSCI

V0 IMS

Subsystems and CSCIs: DMS (Cont.)DDICT Architecture and Interfaces

177625-EMD-002, Rev. 02

DMS

DDICT


ASE(Sybase)


EcDmDictServer

V0 GTWAY CSCI

V0 IMS


Request attribute mappings

Attribute mappings

178625-EMD-002, Rev. 02

DMS

DDICT


ASE(Sybase)


EcDmDictServer

V0 GTWAY CSCI

V0 IMS


Retrieved table data

Update tables

179625-EMD-002, Rev. 02

DMS

DDICT


ASE(Sybase)


EcDmDictServer

V0 GTWAY CSCI

V0 IMS


Retrieve DDICT data

Update DDICT data

180625-EMD-002, Rev. 02

DMS

DDICT


ASE(Sybase)


EcDmDictServer

V0 GTWAY CSCI

V0 IMS


Export valids

181625-EMD-002, Rev. 02

DMS

DDICT


ASE(Sybase)


EcDmDictServer

V0 GTWAY CSCI

V0 IMS


Send/ Replace ESDT installation information

182625-EMD-002, Rev. 02


DMS

DDICT


ASE(Sybase)


EcDmDictServer

V0 GTWAY CSCI

V0 IMS

Request communications support

Validate subscription qualifiers

183625-EMD-002, Rev. 02


DMS

DDICT


ASE(Sybase)


EcDmDictServer

V0 GTWAY CSCI

V0 IMS

Request Management Services

184625-EMD-002, Rev. 02

Subsystems and CSCIs: DMS (Cont.)

• Version Zero Gateway (V0 GTWAY) CSCI– Provides one-way interoperability with the V0 Information

Management System (IMS) for inventory searches, browse requests, product orders, and price estimate requests; search results include links to URLs for Data Pool products

– At the LP DAAC, transmits ASTER product requests to the V0 IMS to allow billing by the billing and accounting system- Distributed Ordering, Reporting, Researching, and Accounting

Network (DORRAN)– Queries between V0 IMS and the ECS V0 GTWAY use the

Object Description Language (ODL) format– One component

- V0 to ECS Gateway Server - allows use of the EOS Data Gateway Web Client to search and request data and services defined within ECS

185625-EMD-002, Rev. 02

Subsystems and CSCIs: DMS (Cont.)V0 GTWAY Architecture & Interfaces

DMS

V0 GTWAY

DDICT CSCI

EcDmV0ToEcsGateway

V0 IMS

EXTERNAL

SUBSETTE

R

SubsetterLocation(e.g., DAAC)

O

186625-EMD-002, Rev. 02

DMS

V0 GTWAY

DDICT CSCI

EcDmV0ToEcsGateway

V0 IMS

EXTERNAL

SUBSETTE

R


O


Request Attribute Mappings

Attribute Mappings

187625-EMD-002, Rev. 02

DMS

V0 GTWAY

DDICT CSCI

EcDmV0ToEcsGateway

V0 IMS

EXTERNAL

SUBSETTE

R


O


Search inventory, request Browse data, product Acquire requests, request price estimate, request subsetted product, request on-demand product

Return product metadata, Browse data, price estimate, product request status, and URLs and expiration dates for Data Pool data, metadata, and browse

188625-EMD-002, Rev. 02

DMS

V0 GTWAY

DDICT CSCI

EcDmV0ToEcsGateway

V0 IMS

EXTERNAL

SUBSETTE

R


O


Status of User Billing

Informa-tion

Product

Requests

requiring billin

g

and accounting

189625-EMD-002, Rev. 02

DMS

V0 GTWAY

DDICT CSCI

EcDmV0ToEcsGateway

V0 IMS

EXTERNAL

SUBSETTE

R


O


Request input product for subsetting; request order status update Product request

acknowledgement; order status acknowledgement

190625-EMD-002, Rev. 02

DMS

V0 GTWAY

DDICT CSCI

EcDmV0ToEcsGateway

V0 IMS

EXTERNAL

SUBSETTE

R


O


Search Inventory,

Request Browse

Data; Product

Acquire requests

for input data for

subsetting

191625-EMD-002, Rev. 02

DMS

V0 GTWAY

DDICT CSCI

EcDmV0ToEcsGateway

V0 IMS

EXTERNAL

SUBSETTE

R


O


Return request

status

Request On-Demand

Product

192625-EMD-002, Rev. 02

DMS

V0 GTWAY

DDICT CSCI

EcDmV0ToEcsGateway

V0 IMS

EXTERNAL

SUBSETTE

R


O


Request metadata

URLs, science data

URLs, browse URLs, and

expiration datesReturn metadata

URLs, science data

URLs, browse

URLs, and expiration dates

193625-EMD-002, Rev. 02

DMS

V0 GTWAY

DDICT CSCI

EcDmV0ToEcsGateway

V0 IMS

EXTERNAL

SUBSETTE

R


O


Product Acquire Requests

194625-EMD-002, Rev. 02

DMS

V0 GTWAY

DDICT CSCI

EcDmV0ToEcsGateway

V0 IMS

EXTERNAL

SUBSETTE

R


O


Request

communications

support

195625-EMD-002, Rev. 02

DMS

V0 GTWAY

DDICT CSCI

EcDmV0ToEcsGateway

V0 IMS

EXTERNAL

SUBSETTE

R


O


Request Management Services

(e.g., order/request tracking

update, User Profile request)

196625-EMD-002, Rev. 02


DataData Pool

(DPL)DATAPOOL

ElectronicProducts


(DPS) Data Proc

0110001011001011001011010111010110001011

InventorySearch


Archive


SDSRVDDIST

STMGT

Inven-toryDB

Collection

Insertand

Retrieve


PUSH


PlanningSubsystem

(PLS) Planning

abc

def

ghi

jkl

mno

PULLSystem Mgmt

SubsystemMSS

AcquireRequests

ProductOrders



Meta-data

ProductRequests

EMOS


Searchand

Access


Data MgmtSubsystem

(DMS)


Searchand

Download





Data

DataIngest


Subsystems and CSCIs: DMS (Cont.)

Data Dictionary(DDICT) CSCI

Data DictionaryMaintenance Tool

EcDmMaintTool

Data DictionaryASE Server


Data DictionaryServer

EcDmDictService

Version Zero Gateway(V0 GTWAY) CSCI

V0 to ECSGateway ServerEcDmV0ToECSGateway

Data MgmtSubsystem

(DMS)

197625-EMD-002, Rev. 02


DataData Pool

(DPL)DATAPOOL

ElectronicProducts


(DPS) Data Proc

0110001011001011001011010111010110001011

InventorySearch


Archive


SDSRVDDIST

STMGT

Inven-toryDB

Collection

Insertand

Retrieve


PUSH


PlanningSubsystem

(PLS) Planning

abc

def

ghi

jkl

mno

PULLSystem Mgmt

SubsystemMSS

AcquireRequests

ProductOrders



Meta-data

ProductRequests

EMOS


Searchand

Access


Data MgmtSubsystem

(DMS)


Searchand

Download





Data

DataIngest


Subsystems and CSCIs: OMS


(OMS)

198625-EMD-002, Rev. 02


• Order Manager Subsystem (OMS)– Manages orders arriving via the following routes:

- DMS V0 Gateway (i.e., from EDG, ECHO, and GDS users)- Spatial Subscription Server (NBSRV)- CSS Machine-to-Machine Gateway- SCLI- Data conversion (HEG) requests or media orders from the Data Pool

Web Access GUI– Provides an Order Manager Server that receives the data

distribution orders and forwards them to the appropriate ECS services- Orders for electronic distribution either filled from the Data Pool or

sent to SDSRV (as appropriate)- Orders for media distribution sent to PDS

– Provides a Graphical User Interface (GUI) for monitoring and controlling Order Manager operations


(OMS)

199625-EMD-002, Rev. 02


• Order Manager Subsystem (OMS) (Cont.)– Uses several COTS tools: RogueWave class libraries, Sybase

Software Developer’s Kit (formerly Open Client), Sybase ASE Server


(OMS)

200625-EMD-002, Rev. 02

Subsystems and CSCIs: OMS (Cont.)

• Order Manager Server (OMSRV) CSCI– Receives Product Distribution Requests from the various

sources– Immediately stores request information in a relational Database

Management System (DBMS)– Validates the requests for correctness (e.g., request size, media

capacity, accessibility, validity of UR)– Queues HEG requests and dispatches individual line items to

HEG services, which subset the individual line items– If the media type or ESDTs of a request are configured for

Synergy III processing mode or the order manager server is configured for Synergy III mode, dispatches each validated request to the appropriate ECS order-fulfillment service; e.g., PDS or SDSRV, depending on whether the request is for physical media or electronic distribution

201625-EMD-002, Rev. 02


• Order Manager Server (OMSRV) CSCI (Cont.)– Otherwise, stages the order to Data Pool storage (and creates

links from staged files to the FtpPull directory in the Data Pool storage if the distribution type is FtpPull), distributes the order to the appropriate service (i.e., Production Module or OMS Ftp Driver) depending on whether distribution type is media or FtpPush, then sends a Distribution Notice to the end user when the order is considered shipped

– Fills valid electronic distribution requests from DPL or submitsvalid requests to SDSRV or PDS (as appropriate)

– Generates Operator Interventions for invalid requests– Two major components

- Order Manager Server - interacts with the Order Manager Database, Order Manager Ftp Push Server, Data Pool, Product Distribution System, and Science Data Server

- Order Manager ASE Server - COTS database server

202625-EMD-002, Rev. 02


• Order Manager GUI (OMGUI) CSCI– Based on web standards– Communicates directly with the Order Manager Service

database (not through a server)– Permits monitoring and control of the OMSRV (e.g., view

operator alerts, view status of queues, suspend and resume queues, view and modify configuration parameters, monitor statistics)

– Monitor open Operator Interventions for invalid requests; manage interventions; view closed interventions; view, cancel, suspend, resume, or resubmit distribution requests; view order information and user profile data; view, update, or cancel bundling orders

– One major component- Order Manager GUI - interacts with the Order Manager Database

203625-EMD-002, Rev. 02


• Production Module CSCI– Processes physical media distribution (PMD) requests

(Synergy V feature)– One major component

- EcOmPdMediaProduction - processes product distribution requests on physical media

204625-EMD-002, Rev. 02

Subsystems and CSCIs: OMS (Cont.)OMS Architecture and Interface

OMS

OMSRVASE(Sybase)

OMS Database

EcOmOrderManager

OMGUI

EcOmGui

Production ModuleEcOmPdMediaProduction

205625-EMD-002, Rev. 02

OMS

OMSRVASE(Sybase)

OMS Database

EcOmOrderManager

OMGUI

EcOmGui



Insert Product Distribution Request (EcNbActionDriver.pl)

Insert Product Distribution Request (EcCsMtMGateway)

206625-EMD-002, Rev. 02

OMS

OMSRVASE(Sybase)

OMS Database

EcOmOrderManager

OMGUI

EcOmGui



Granule staging request

Insert Hard Media Distribution Request or HEG Request (EcDlWebAccess)

Insert Product Distribution Request (EcDmV0ToEcsGateway)

207625-EMD-002, Rev. 02

OMS

OMSRVASE(Sybase)

OMS Database

EcOmOrderManager

OMGUI

EcOmGui



Create/update Product Distribution Request; create Operator Intervention; create Alert; submit granule staging request

Retrieve Distribution Request; retrieve configuration information; retrieve e-mail information

208625-EMD-002, Rev. 02

OMS

OMSRVASE(Sybase)

OMS Database

EcOmOrderManager

OMGUI

EcOmGui



Order information; user profile information

209625-EMD-002, Rev. 02

OMS

OMSRVASE(Sybase)

OMS Database

EcOmOrderManager

OMGUI

EcOmGui



Retrieve OM Server information, Inter-vention Requests, and PMD actions

Update OM Server configuration data; update/respond to Intervention Requests; control PMD operations

210625-EMD-002, Rev. 02

OMS

OMSRVASE(Sybase)

OMS Database

EcOmOrderManager

OMGUI

EcOmGui



Pre-Synergy IV: Submit electronic distribution request

Synergy IV/V: FtpPush request

211625-EMD-002, Rev. 02

OMS

OMSRVASE(Sybase)

OMS Database

EcOmOrderManager

OMGUI

EcOmGui



Pre-Syn

ergy V

:

Dispatc

h har

d

media

distrib

ution

reque

st

Synerg

y V:

Dispatc

h phys

ical

media

distrib

ution

reque

st

Retrieve PMD request informa-tion; update PMD request information

212625-EMD-002, Rev. 02

OMS

OMSRVASE(Sybase)

OMS Database

EcOmOrderManager

OMGUI

EcOmGui



Submit HEG request

213625-EMD-002, Rev. 02

OMS

OMSRVASE(Sybase)

OMS Database

EcOmOrderManager

OMGUI

EcOmGui



Request commun-ications support (e.g., send e-mail)

214625-EMD-002, Rev. 02


DataData Pool

(DPL)DATAPOOL

ElectronicProducts


(DPS) Data Proc

0110001011001011001011010111010110001011

InventorySearch


Archive


SDSRVDDIST

STMGT

Inven-toryDB

Collection

Insertand

Retrieve


PUSH


PlanningSubsystem

(PLS) Planning

abc

def

ghi

jkl

mno

PULLSystem Mgmt

SubsystemMSS

AcquireRequests

ProductOrders



Meta-data

ProductRequests

EMOS


Searchand

Access


Data MgmtSubsystem

(DMS)


Searchand

Download





Data

DataIngest



Order Manager GUI(OMGUI) CSCI

Order Manager GUIEcOmGui

Order Manager Server(OMSRV) CSCI

Order ManagerServer

EcOmOrderManager

Order ManagerASE Server



(OMS)

Production Module CSCI

ProductionModule

EcOmPdMediaProduction

215625-EMD-002, Rev. 02


DataData Pool

(DPL)DATAPOOL

ElectronicProducts


(DPS) Data Proc

0110001011001011001011010111010110001011

InventorySearch


Archive


SDSRVDDIST

STMGT

Inven-toryDB

Collection

Insertand

Retrieve


PUSH


PlanningSubsystem

(PLS) Planning

abc

def

ghi

jkl

mno

PULLSystem Mgmt

SubsystemMSS

AcquireRequests

ProductOrders



Meta-data

ProductRequests

EMOS


Searchand

Access


Data MgmtSubsystem

(DMS)


Searchand

Download

No more PDS Spatial Subscrip-tion Server



Data

DataIngest


Subsystems and CSCIs: PLS

PlanningSubsystem

(PLS) Planning

abc

def

ghi

jkl

mno

216625-EMD-002, Rev. 02

Subsystems and CSCIs: PLS

• Planning Subsystem (PLS)– Allows operations staff to define

data processing tasks to be performed at a site– Generates efficient plans for scheduling defined data

processing and reprocessing tasks according to production rules that define how a Product Generation Executive (PGE) is to run

– Coordinates production with the Data Server and Data Processing subsystems to achieve a highly automated production system

– Interfaces with the Algorithm Integration and Test Tools CSCI within DPS for information on PGEs

– Permits entry of Production Requests and generates resulting Data Processing Requests (DPRs)

– Uses a set of Raytheon-provided COTS libraries as a basis for its scheduling components (Resource Planning Workbench and Production Planning Workbench)

PlanningSubsystem Planning

abc

def

ghi

jkl

mno

217625-EMD-002, Rev. 02

Subsystems and CSCIs: PLS (Cont.)

• Production Planning (PLANG) CSCI– Eight major components

- Data Store - handles insertion of data for planning and processing activities into the PDPS shared database

- Resource Planning Workbench - GUIs for preparing a site resource schedule [Resource Editor (EcPlRpRe), Scheduling Interface (EcPlRpSi), Timeline (EcPlRpTl)]

- Production Request Editor - GUI for submitting production requests that describe the data products to be produced; uses PGE descriptions to generate the DPRs necessary to meet the requests(EcPlPREditor_IF)

- Production Planning Workbench - GUIs and a script for preparing a site production schedule [Workbench (EcPlWb), Timeline (EcPlTl) and Chain Determination (EcPlDetermineChain.pl)]

- On-Demand Production Request Manager - receives requests for data from the scientist via the EDG web page, generates the necessary Production Request, submits it for processing, and distributes the data to the scientist (EcPlOdMgr)

218625-EMD-002, Rev. 02


• Production Planning (PLANG) CSCI (Cont.)– Eight major components (Cont.)

- Production Request Generator - command-line interface for creating production requests (EcPlPRGenerator)

- Subscription Manager - server to manage receipt of subscription notifications from the Data Server via SBSRV (EcPlSubMgr)

- Production Strategies GUI - used to create a set of planning priorities to be applied to each DPR in a plan (EcPlProdStrat)

219625-EMD-002, Rev. 02

PLSPLANG

EcPlSubMgr

EcPlPREditor_IF

EcPlRpRe

EcPlRpSi

EcPlTl

EcPlProdStratEcPlRpTl

PDPSDatabase

ASE(Sybase)

EcPlWbEcPlOdMgr

EcPlPRGenerator

EcPlDetermineChain.pl

Subsystems and CSCIs: PLS (Cont.)PLANG Architecture and Interfaces

220625-EMD-002, Rev. 02

PLSPLANG

EcPlSubMgr

EcPlPREditor_IF

EcPlRpRe

EcPlRpSi

EcPlTl


PDPSDatabase

ASE(Sybase)

EcPlWbEcPlOdMgr

EcPlPRGenerator



Request Management Support

221625-EMD-002, Rev. 02

PLSPLANG

EcPlSubMgr

EcPlPREditor_IF

EcPlRpRe

EcPlRpSi

EcPlTl


PDPSDatabase

ASE(Sybase)

EcPlWbEcPlOdMgr

EcPlPRGenerator



Submit On-Demand Product request; Submit Standing On-Demand orders (from V0 Gateway)

222625-EMD-002, Rev. 02

PLSPLANG

EcPlSubMgr

EcPlPREditor_IF

EcPlRpRe

EcPlRpSi

EcPlTl


PDPSDatabase

ASE(Sybase)

EcPlWbEcPlOdMgr

EcPlPRGenerator



Tracking elements (e.g., order ID,request ID)

Request On-Demand tracking elements; request standing order tracking elements

223625-EMD-002, Rev. 02

PLSPLANG

EcPlSubMgr

EcPlPREditor_IF

EcPlRpRe

EcPlRpSi

EcPlTl


PDPSDatabase

ASE(Sybase)

EcPlWbEcPlOdMgr

EcPlPRGenerator



Search inventory; submit Acquire

224625-EMD-002, Rev. 02

PLSPLANG

EcPlSubMgr

EcPlPREditor_IF

EcPlRpRe

EcPlRpSi

EcPlTl


PDPSDatabase

ASE(Sybase)

EcPlWbEcPlOdMgr

EcPlPRGenerator



Subscribe

225625-EMD-002, Rev. 02

PLSPLANG

EcPlSubMgr

EcPlPREditor_IF

EcPlRpRe

EcPlRpSi

EcPlTl


PDPSDatabase

ASE(Sybase)

EcPlWbEcPlOdMgr

EcPlPRGenerator



Store/Update PRs

Retrieve PRs

226625-EMD-002, Rev. 02

PLSPLANG

EcPlSubMgr

EcPlPREditor_IF

EcPlRpRe

EcPlRpSi

EcPlTl


PDPSDatabase

ASE(Sybase)

EcPlWbEcPlOdMgr

EcPlPRGenerator



Submit DPR

227625-EMD-002, Rev. 02

PLSPLANG

EcPlSubMgr

EcPlPREditor_IF

EcPlRpRe

EcPlRpSi

EcPlTl


PDPSDatabase

ASE(Sybase)

EcPlWbEcPlOdMgr

EcPlPRGenerator



Search inventory; Query for metadata attributes

228625-EMD-002, Rev. 02

PLSPLANG

EcPlSubMgr

EcPlPREditor_IF

EcPlRpRe

EcPlRpSi

EcPlTl


PDPSDatabase

ASE(Sybase)

EcPlWbEcPlOdMgr

EcPlPRGenerator



Subscribe

229625-EMD-002, Rev. 02

PLSPLANG

EcPlSubMgr

EcPlPREditor_IF

EcPlRpRe

EcPlRpSi

EcPlTl


PDPSDatabase

ASE(Sybase)

EcPlWbEcPlOdMgr

EcPlPRGenerator



Create/Store PRs;

Create/Store DPRs;

Delete PRs;

Delete DPRs

Retrieve PRs;

Retrieve DPRs

230625-EMD-002, Rev. 02

PLSPLANG

EcPlSubMgr

EcPlPREditor_IF

EcPlRpRe

EcPlRpSi

EcPlTl


PDPSDatabase

ASE(Sybase)

EcPlWbEcPlOdMgr

EcPlPRGenerator



Search inventory; Query for metadata attributes

231625-EMD-002, Rev. 02

PLSPLANG

EcPlSubMgr

EcPlPREditor_IF

EcPlRpRe

EcPlRpSi

EcPlTl


PDPSDatabase

ASE(Sybase)

EcPlWbEcPlOdMgr

EcPlPRGenerator



Subscribe

232625-EMD-002, Rev. 02

PLSPLANG

EcPlSubMgr

EcPlPREditor_IF

EcPlRpRe

EcPlRpSi

EcPlTl


PDPSDatabase

ASE(Sybase)

EcPlWbEcPlOdMgr

EcPlPRGenerator



Create/Store PRs;Create/Store DPRs;

233625-EMD-002, Rev. 02

PLSPLANG

EcPlSubMgr

EcPlPREditor_IF

EcPlRpRe

EcPlRpSi

EcPlTl


PDPSDatabase

ASE(Sybase)

EcPlWbEcPlOdMgr

EcPlPRGenerator



Submit DPR

234625-EMD-002, Rev. 02

PLSPLANG

EcPlSubMgr

EcPlPREditor_IF

EcPlRpRe

EcPlRpSi

EcPlTl


PDPSDatabase

ASE(Sybase)

EcPlWbEcPlOdMgr

EcPlPRGenerator



Retrieve productionstrategiesCreate/update production strategies

235625-EMD-002, Rev. 02

PLSPLANG

EcPlSubMgr

EcPlPREditor_IF

EcPlRpRe

EcPlRpSi

EcPlTl


PDPSDatabase

ASE(Sybase)

EcPlWbEcPlOdMgr

EcPlPRGenerator



Retrieve plans

Store/Update plans

236625-EMD-002, Rev. 02

PLSPLANG

EcPlSubMgr

EcPlPREditor_IF

EcPlRpRe

EcPlRpSi

EcPlTl


PDPSDatabase

ASE(Sybase)

EcPlWbEcPlOdMgr

EcPlPRGenerator



DPR scheduling information

237625-EMD-002, Rev. 02

PLSPLANG

EcPlSubMgr

EcPlPREditor_IF

EcPlRpRe

EcPlRpSi

EcPlTl


PDPSDatabase

ASE(Sybase)

EcPlWbEcPlOdMgr

EcPlPRGenerator



Submit/cancel DPR; Submit ground event

Update chaininfoRequest chain

determination

238625-EMD-002, Rev. 02

PLSPLANG

EcPlSubMgr

EcPlPREditor_IF

EcPlRpRe

EcPlRpSi

EcPlTl


PDPSDatabase

ASE(Sybase)

EcPlWbEcPlOdMgr

EcPlPRGenerator



Subscriptionnotification

239625-EMD-002, Rev. 02

PLSPLANG

EcPlSubMgr

EcPlPREditor_IF

EcPlRpRe

EcPlRpSi

EcPlTl


PDPSDatabase

ASE(Sybase)

EcPlWbEcPlOdMgr

EcPlPRGenerator



Retrieve information relating jobs to subscriptions

Update with data availability notification

240625-EMD-002, Rev. 02

PLSPLANG

EcPlSubMgr

EcPlPREditor_IF

EcPlRpRe

EcPlRpSi

EcPlTl


PDPSDatabase

ASE(Sybase)

EcPlWbEcPlOdMgr

EcPlPRGenerator



Metadata attribute

query (to determine

which jobs can use

an available

granule as input)

241625-EMD-002, Rev. 02

PLSPLANG

EcPlSubMgr

EcPlPREditor_IF

EcPlRpRe

EcPlRpSi

EcPlTl


PDPSDatabase

ASE(Sybase)

EcPlWbEcPlOdMgr

EcPlPRGenerator



Release DPR

242625-EMD-002, Rev. 02

PLSPLANG

EcPlSubMgr

EcPlPREditor_IF

EcPlRpRe

EcPlRpSi

EcPlTl


PDPSDatabase

ASE(Sybase)

EcPlWbEcPlOdMgr

EcPlPRGenerator



Delete granules (for cancelled DPR)

243625-EMD-002, Rev. 02

PLSPLANG

EcPlSubMgr

EcPlPREditor_IF

EcPlRpRe

EcPlRpSi

EcPlTl


PDPSDatabase

ASE(Sybase)

EcPlWbEcPlOdMgr

EcPlPRGenerator



Store/Update resource definitions

Retrieve resource definitions

244625-EMD-002, Rev. 02

PLSPLANG

EcPlSubMgr

EcPlPREditor_IF

EcPlRpRe

EcPlRpSi

EcPlTl


PDPSDatabase

ASE(Sybase)

EcPlWbEcPlOdMgr

EcPlPRGenerator



Define/Update ground events

Retrieve ground events

245625-EMD-002, Rev. 02

PLSPLANG

EcPlSubMgr

EcPlPREditor_IF

EcPlRpRe

EcPlRpSi

EcPlTl


PDPSDatabase

ASE(Sybase)

EcPlWbEcPlOdMgr

EcPlPRGenerator



Cancel ground event

246625-EMD-002, Rev. 02

PLSPLANG

EcPlSubMgr

EcPlPREditor_IF

EcPlRpRe

EcPlRpSi

EcPlTl


PDPSDatabase

ASE(Sybase)

EcPlWbEcPlOdMgr

EcPlPRGenerator



Resource reservation schedule

247625-EMD-002, Rev. 02


DataData Pool

(DPL)DATAPOOL

ElectronicProducts


(DPS) Data Proc

0110001011001011001011010111010110001011

InventorySearch


Archive


SDSRVDDIST

STMGT

Inven-toryDB

Collection

Insertand

Retrieve


PUSH


PlanningSubsystem

(PLS) Planning

abc

def

ghi

jkl

mno

PULLSystem Mgmt

SubsystemMSS

AcquireRequests

ProductOrders



Meta-data

ProductRequests

EMOS


Searchand

Access


Data MgmtSubsystem

(DMS)


Searchand

Download





Data

DataIngest


PlanningSubsystem

(PLS) Planning

abc

def

ghi

jkl

mno


Production Planning(PLANG) CSCI

DataStore

Sybase Database (COTS)

Resource PlanningWorkbench

EcPlRpReEcPlRpSiEcPlRpTl

ProductionRequest Editor

EcPlPrEditor_IF

ProductionPlanning Workbench

EcPlWbEcPlTl


On-DemandProduction RequestManager (ODPRM)

EcPlOdMgr

SubscriptionManagerEcPlSubMgr

ProductionStrategies GUI

EcPlProdStrat

ProductionRequest Generator

EcPlPRGenerator

248625-EMD-002, Rev. 02

Subsystems and CSCIs: DPS


DataData Pool

(DPL)DATAPOOL

ElectronicProducts


(DPS) Data Proc

0110001011001011001011010111010110001011

InventorySearch


Archive


SDSRVDDIST

STMGT

Inven-toryDB

Collection

Insertand

Retrieve


PUSH


PlanningSubsystem

(PLS) Planning

abc

def

ghi

jkl

mno

PULLSystem Mgmt

SubsystemMSS

AcquireRequests

ProductOrders



Meta-data

ProductRequests

EMOS


Searchand

Access


Data MgmtSubsystem

(DMS)


Searchand

Download





Data

DataIngest



(DPS) Data Proc

0110001011001011001011010111010110001011

249625-EMD-002, Rev. 02

Subsystems and CSCIs: DPS

• Data Processing Subsystem (DPS)– Manages allocation and recovery of computer resources (e.g.,

CPU, disk space) used in processing science data– Manages, queues, and executes DPRs– Supports execution of science algorithms through the Science

Data Processing (SDP) Toolkit– Supports preliminary processing of ancillary data sets– Provides an Algorithm Integration and Test (AIT) environment

for the introduction of science software– Provides a Quality Assessment (QA) environment for updating

the quality flags in metadata for data products

DataProcessingSubsystem Data Proc

0110001011001011001011010111010110001011

250625-EMD-002, Rev. 02

Subsystems and CSCIs: DPS (Cont.)

• Data Processing Subsystem (DPS) (Cont.)– Uses COTS tools

- AutoSys: a job scheduling software application to automate operations in a distributed UNIX environment

- Job Management Web Interfaces: provide interfaces to monitor and manage the job schedule being processed in AutoSys

- Sybase: ASE server

251625-EMD-002, Rev. 02


• Processing (PRONG) CSCI– Provides services required to manage and monitor the Science

Data Processing environment, which executes Science Software items (PGEs) and produces data products

– Nine major components- Job Management - handles flow of information to the COTS

products; also creates and starts Ground Event jobs- Execution Management - initiates execution of PGEs and performs

final activities subsequent to execution of PGEs; handles flow of science data to and from science processing resources (through adata management software library, DpPrDM); also provides status of On-demand Processing requests

- PGE Management - controls and monitors execution of PGEs and the growth of the output products (EcDpPrRunPGE); measures and reports resource use to AutoSys (EcDpPrRusage)

252625-EMD-002, Rev. 02


• Processing (PRONG) CSCI (Cont.)– Nine major components (Cont.)

- Deletion Server - notifies Science Data Server to remove interim granules that are no longer needed

- Quality Assurance Monitor - supports visualizing science data products and updating QA metadata

- Data Preprocessing - manages preprocessing of ancillary data used as inputs to a PGE

- AutoSys - provides the job scheduling engine (COTS)- Data Store - handles insertion of data for planning and processing

activities into the PDPS shared database- Ground Event Process - initiated by Job Management upon receipt

of a ground event request; sets a computer resource to an off-line state, making it unavailable for PGEs during the request

253625-EMD-002, Rev. 02

DPSPRONG

PDPSDatabase

ASE(Sybase)

EcDpPrJobMgmtEcDpPrJobMgmtClient

EcDpPrDeletion

AutoSys

EcDpPrEM EcDpPrQaMonitorGUI

EcDpPrRunPGEEcDpPrRusage

PGEs (e.g., EcDpPrAm1EdosEphAttDPREP_PGE,EcDpPrAm1FddAttitudeDPREP_PGE,EcDpPrAm1FddEphemerisDPREP_PGE)

EcDpPrViewJobStates

EcDpPrGE

User

Subsystems and CSCIs: DPS (Cont.)PRONG Architecture and Interfaces

254625-EMD-002, Rev. 02

DPSPRONG

PDPSDatabase

ASE(Sybase)


EcDpPrDeletion

AutoSys




EcDpPrViewJobStates

EcDpPrGE

User


Submit requests to create, release, cancel DPR jobs (and ground events)

255625-EMD-002, Rev. 02

DPSPRONG

PDPSDatabase

ASE(Sybase)


EcDpPrDeletion

AutoSys




EcDpPrViewJobStates

EcDpPrGE

User


Create, release, cancel job box; create/cancel ground event job

256625-EMD-002, Rev. 02

DPSPRONG

PDPSDatabase

ASE(Sybase)


EcDpPrDeletion

AutoSys




EcDpPrViewJobStates

EcDpPrGE

User


InitiateExecution

257625-EMD-002, Rev. 02

DPSPRONG

PDPSDatabase

ASE(Sybase)


EcDpPrDeletion

AutoSys




EcDpPrViewJobStates

EcDpPrGE

User


Initiate execu-tion of a job

258625-EMD-002, Rev. 02

DPSPRONG

PDPSDatabase

ASE(Sybase)


EcDpPrDeletion

AutoSys




EcDpPrViewJobStates

EcDpPrGE

User


Monitorstatus

259625-EMD-002, Rev. 02

DPSPRONG

PDPSDatabase

ASE(Sybase)


EcDpPrDeletion

AutoSys




EcDpPrViewJobStates

EcDpPrGE

User


Display Job States

260625-EMD-002, Rev. 02

DPSPRONG

PDPSDatabase

ASE(Sybase)


EcDpPrDeletion

AutoSys




EcDpPrViewJobStates

EcDpPrGE

User


Request/Allocate disk space; update PGE information, request CPU and RAM allocation

261625-EMD-002, Rev. 02

DPSPRONG

PDPSDatabase

ASE(Sybase)


EcDpPrDeletion

AutoSys




EcDpPrViewJobStates

EcDpPrGE

User


Request MCF;

insert granules;

request PGE tar

file; insert product

history tar files;

insert Failed PGE

tar file

Granule acquires; PGE tar file;

MCF

262625-EMD-002, Rev. 02

DPSPRONG

PDPSDatabase

ASE(Sybase)


EcDpPrDeletion

AutoSys




EcDpPrViewJobStates

EcDpPrGE

User


Retrieve data,

granule information

Update data,

granule information

263625-EMD-002, Rev. 02

DPSPRONG

PDPSDatabase

ASE(Sybase)


EcDpPrDeletion

AutoSys




EcDpPrViewJobStates

EcDpPrGE

User


Updatemetadata (QA flag)

Retrieve metadata; Retrieve Product History

264625-EMD-002, Rev. 02

DPSPRONG

PDPSDatabase

ASE(Sybase)


EcDpPrDeletion

AutoSys




EcDpPrViewJobStates

EcDpPrGE

User


Request (interim) granule deletion

265625-EMD-002, Rev. 02

DPSPRONG

PDPSDatabase

ASE(Sybase)


EcDpPrDeletion

AutoSys




EcDpPrViewJobStates

EcDpPrGE

User


Delete granules (associated with cancelled DPR)

266625-EMD-002, Rev. 02

DPSPRONG

PDPSDatabase

ASE(Sybase)


EcDpPrDeletion

AutoSys




EcDpPrViewJobStates

EcDpPrGE

User


Send On-Demand Failure Message

267625-EMD-002, Rev. 02

DPSPRONG

PDPSDatabase

ASE(Sybase)


EcDpPrDeletion

AutoSys




EcDpPrViewJobStates

EcDpPrGE

User


Update Request

Status

268625-EMD-002, Rev. 02

DPSPRONG

PDPSDatabase

ASE(Sybase)


EcDpPrDeletion

AutoSys




EcDpPrViewJobStates

EcDpPrGE

User


Request

Communications

Support

269625-EMD-002, Rev. 02

DPSPRONG

PDPSDatabase

ASE(Sybase)


EcDpPrDeletion

AutoSys




EcDpPrViewJobStates

EcDpPrGE

User


Management Services

270625-EMD-002, Rev. 02


• Algorithm Integration and Test Tools (AITTL) CSCI– Provides a set of tools used for testing and integration of new

science software, new versions of science software, and user methods into the Science Data Processing operational environment

– Combines custom-developed code with COTS software– Tools are accessed from a centralized application called the

Science Software Integration and Test (SSIT) Manager

271625-EMD-002, Rev. 02


• Algorithm Integration and Test Tools (AITTL) CSCI (Cont.)– Six major components

- Science Software Archive Package (SSAP) GUI - allows for the creation, update, and deletion of SSAPs

- SSIT Manager - GUI for SSIT activities; provides menus to launch other SSIT applications and a checklist to mark completion of SSIT functions

- Define PGE - a group of applications to specify a PGE in the PDPS database

- View/Compare Tools - a group of applications for viewing and comparing data files

- Check Software Tools - a group of applications that check the source code for PGEs and their process control files (PCFs) for errors or prohibited functions

- Insert/Acquire Tools - a group of applications that provide mechanisms to insert and acquire data items from Data Server

272625-EMD-002, Rev. 02DPSAITTL

ASE (Sybase)PDPSDatabase

EcDpAtSSAPGui

PGE/ESDTODLFiles

Define

PGE

View/Compare

EcDpAtBinDiffGui

EcDpAtMgrXdiff

EcDpAtCheckHdfFile

EcDpAtRegisterPGE

EcDpAtCheckODL

EcDpAtCreateODLTemplate

EcDpAtOpDbGui

EcDpAtMgr(SSIT Manager)

EcDpAtStageDAP

EcDpAtInsertTestFile

EcDpAtInsertExeTarFile

EcDpAtGetMCF

Insert/Acquire

EcDpAtInsertStaticFile

EcDpAtMgrLogDump

EcDpAtCheckPCF

EcDpAtPrologs

EcDpAtBadFuncGui

EcDpAtBadFunc

CheckSoftware

Subsystems and CSCIs: DPS (Cont.)AITTL Architecture and Interfaces



EcDpAtSSAPGui

PGE/ESDTODLFiles

Define

PGE

View/Compare

EcDpAtBinDiffGui

EcDpAtMgrXdiff

EcDpAtCheckHdfFile

EcDpAtRegisterPGE

EcDpAtCheckODL


EcDpAtOpDbGui


EcDpAtStageDAP



EcDpAtGetMCF

Insert/Acquire


EcDpAtMgrLogDump

EcDpAtCheckPCF

EcDpAtPrologs

EcDpAtBadFuncGui

EcDpAtBadFunc

CheckSoftware


Initiate Execution



EcDpAtSSAPGui

PGE/ESDTODLFiles

Define

PGE

View/Compare

EcDpAtBinDiffGui

EcDpAtMgrXdiff

EcDpAtCheckHdfFile

EcDpAtRegisterPGE

EcDpAtCheckODL


EcDpAtOpDbGui


EcDpAtStageDAP



EcDpAtGetMCF

Insert/Acquire


EcDpAtMgrLogDump

EcDpAtCheckPCF

EcDpAtPrologs

EcDpAtBadFuncGui

EcDpAtBadFunc

CheckSoftware


Request/insert SSAP information

SSAP information



EcDpAtSSAPGui

PGE/ESDTODLFiles

Define

PGE

View/Compare

EcDpAtBinDiffGui

EcDpAtMgrXdiff

EcDpAtCheckHdfFile

EcDpAtRegisterPGE

EcDpAtCheckODL


EcDpAtOpDbGui


EcDpAtStageDAP



EcDpAtGetMCF

Insert/Acquire


EcDpAtMgrLogDump

EcDpAtCheckPCF

EcDpAtPrologs

EcDpAtBadFuncGui

EcDpAtBadFunc

CheckSoftware


Request PGE &

data insertion;

Request MCF;

Request DAPs;

Request product



EcDpAtSSAPGui

PGE/ESDTODLFiles

Define

PGE

View/Compare

EcDpAtBinDiffGui

EcDpAtMgrXdiff

EcDpAtCheckHdfFile

EcDpAtRegisterPGE

EcDpAtCheckODL


EcDpAtOpDbGui


EcDpAtStageDAP



EcDpAtGetMCF

Insert/Acquire


EcDpAtMgrLogDump

EcDpAtCheckPCF

EcDpAtPrologs

EcDpAtBadFuncGui

EcDpAtBadFunc

CheckSoftware


Insert/Update

Granules and

PGE Information



EcDpAtSSAPGui

PGE/ESDTODLFiles

Define

PGE

View/Compare

EcDpAtBinDiffGui

EcDpAtMgrXdiff

EcDpAtCheckHdfFile

EcDpAtRegisterPGE

EcDpAtCheckODL


EcDpAtOpDbGui


EcDpAtStageDAP



EcDpAtGetMCF

Insert/Acquire


EcDpAtMgrLogDump

EcDpAtCheckPCF

EcDpAtPrologs

EcDpAtBadFuncGui

EcDpAtBadFunc

CheckSoftware


Request

ODL File

Receive

PGE Data



EcDpAtSSAPGui

PGE/ESDTODLFiles

Define

PGE

View/Compare

EcDpAtBinDiffGui

EcDpAtMgrXdiff

EcDpAtCheckHdfFile

EcDpAtRegisterPGE

EcDpAtCheckODL


EcDpAtOpDbGui


EcDpAtStageDAP



EcDpAtGetMCF

Insert/Acquire


EcDpAtMgrLogDump

EcDpAtCheckPCF

EcDpAtPrologs

EcDpAtBadFuncGui

EcDpAtBadFunc

CheckSoftware


Request PGE Insertion or Update



EcDpAtSSAPGui

PGE/ESDTODLFiles

Define

PGE

View/Compare

EcDpAtBinDiffGui

EcDpAtMgrXdiff

EcDpAtCheckHdfFile

EcDpAtRegisterPGE

EcDpAtCheckODL


EcDpAtOpDbGui


EcDpAtStageDAP



EcDpAtGetMCF

Insert/Acquire


EcDpAtMgrLogDump

EcDpAtCheckPCF

EcDpAtPrologs

EcDpAtBadFuncGui

EcDpAtBadFunc

CheckSoftware


Request Communications

Support



EcDpAtSSAPGui

PGE/ESDTODLFiles

Define

PGE

View/Compare

EcDpAtBinDiffGui

EcDpAtMgrXdiff

EcDpAtCheckHdfFile

EcDpAtRegisterPGE

EcDpAtCheckODL


EcDpAtOpDbGui


EcDpAtStageDAP



EcDpAtGetMCF

Insert/Acquire


EcDpAtMgrLogDump

EcDpAtCheckPCF

EcDpAtPrologs

EcDpAtBadFuncGui

EcDpAtBadFunc

CheckSoftware


ManagementServices

281625-EMD-002, Rev. 02


• SDP Toolkit (SDPTK) CSCI– Not described in detail in this course – Provides a set of software tools used to integrate Science

Software into ECS– Provides common functionality (e.g., geolocation) required

across the ECS community– Allows Science Data Processing to support generation of data

products in a heterogeneous computer hardware environment- Facilitates the smooth transition and integration of science

software code into the DAAC by abstracting out science process dependencies on external system architecture

- Provides an interface between science software and the production system environment

- Interface is implemented in both the SCF development environments and DAAC production environments

282625-EMD-002, Rev. 02


• SDP Toolkit (SDPTK) CSCI (Cont.)– Insulates science software from the SDP software and provides

a development environment that emulates critical SDP functions- Helps ensure code portability as the algorithm is ported from

development hardware, through the DAAC system, and through potential hardware changes as ECS matures

- Provides for limited access and control to system level resources, including processes, shared memory, and I/O capabilities

- Where control of system resources is necessary (e.g.,shared memory allocation), Toolkit provides a set of routines through which the application must obtain those services

- Partitioning and layering of operating system services allows Toolkit to work on behalf of DPS in allocating, deallocating, and making use of system-wide shared resources

283625-EMD-002, Rev. 02


– SDP Toolkit Tools - Mandatory- File I/O Tools- Error/Status Reporting [Status

Message File (SMF) Tools]- Process Control Tools- Shared Memory Management

Tools- Bit Manipulation Tools- Spacecraft Ephemeris and

Attitude Data Access Tools- Time and Date Conversion

Tools

– SDP Toolkit Tools - Optional - Digital Elevation Model Tools- Ancillary Data Tools- Celestial Body Position Tools- Coordinate System

Conversion Tools- Geo-Coordinate

Transformation Tools- Math and Statistical Support

Tools- Constants and Unit

Conversions- Dynamic Memory Management

Tools- Graphics Support Tools

• SDP Toolkit (SDPTK) CSCI (Cont.)

284625-EMD-002, Rev. 02


DataData Pool

(DPL)DATAPOOL

ElectronicProducts


(DPS) Data Proc

0110001011001011001011010111010110001011

InventorySearch


Archive


SDSRVDDIST

STMGT

Inven-toryDB

Collection

Insertand

Retrieve


PUSH


PlanningSubsystem

(PLS) Planning

abc

def

ghi

jkl

mno

PULLSystem Mgmt

SubsystemMSS

AcquireRequests

ProductOrders



Meta-data

ProductRequests

EMOS


Searchand

Access


Data MgmtSubsystem

(DMS)


Searchand

Download





Data

DataIngest



(DPS) Data Proc

0110001011001011001011010111010110001011


Algorithm Integration and Test Tools(AITTL) CSCI

SSAP GUIEcDpAtSSAPGui

SSIT ManagerEcDpAtMgr

Define PGEEcDpAtCheckODL

EcDpAtCreateODLTemplateEcDpAtRegisterPGE

EcDpAtOpDbGui

View/CompareData ToolsEcDpAtBinDiffGui

EcDpAtCheckHdfFileEcDpAtMgrXdiff

Check SoftwareTools

EcDpAtCheckPCFEcDpAtBadFuncGui

EcDpAtBadFuncEcDpAtPrologs

Insert/AcquireTools

EcDpAtMgrLogDumpEcDpAtInsertStaticFile

EcDpAtInsertExeTarFileEcDpAtInsertTestFile

EcDpAtStageDAPEcDpAtGetMCF

SDP Toolkit(SDPTK) CSCI

(Separatesoftware libraries

for SSI&T)

Processing(PRONG) CSCI

ExecutionManagement

EcDpPrEMDpPrDM (library)

PGEManagement


DeletionServer

EcDpPrDeletion

Job ManagementEcDpPrJobMgmt

EcDpPrJobMgmtClientEcDpPrViewJobStates

AutoSysCOTS Product

QAMonitor

EcDpPrQaMonitorGUI

DataStore


Ground EventProcess

EcDpPrGE

Data PreprocessingEcDpPrAm1EdosEphAttDPREP_PGEEcDpPrAm1FddAttitudeDPREP_PGE

EcDpPrAm1FddEphemerisDPREP_PGEEcDpPrDumpAttitudeDPREP

EcDpPrDumpEphemerisDPREPEcDpPrPm1FddEphemerisDPREP_PGE

EcDpPrPm1AttitudeDPREP_PGEEcDpPrAuraEphemerisDPREP_PGE

EcDpPrAuraAttitudeDPREP_PGE

285625-EMD-002, Rev. 02


DataData Pool

(DPL)DATAPOOL

ElectronicProducts


(DPS) Data Proc

0110001011001011001011010111010110001011

InventorySearch


Archive


SDSRVDDIST

STMGT

Inven-toryDB

Collection

Insertand

Retrieve


PUSH


PlanningSubsystem

(PLS) Planning

abc

def

ghi

jkl

mno

PULLSystem Mgmt

SubsystemMSS

AcquireRequests

ProductOrders



Meta-data

ProductRequests

EMOS


Searchand

Access


Data MgmtSubsystem

(DMS)


Searchand

Download





Data

DataIngest


Subsystems and CSCIs: MSS


MSS

286625-EMD-002, Rev. 02

Subsystems and CSCIs: MSS

• System Management Subsystem (MSS)– Provides the set of tools needed by

maintenance and operations staff to manage ECS operations– Addresses five areas

- Fault Management- Configuration Management- Accountability Management- Performance Management- Security Management

– Installed locally at each DAAC and at System Monitoring and Coordination Center (SMC)

– Uses COTS applications extensively, including Sybase Replication Server

– Includes ECS Assistant, a GUI that runs an extensive array of UNIX scripts for system installation, monitoring, and administration


(MSS)

287625-EMD-002, Rev. 02

Subsystems and CSCIs: MSS (Cont.)

• Management CSCI (MCI)– Primarily COTS-based, with some custom software– Provides services for monitoring and coordinating ECS– Network and Enterprise Management Framework component

- Whazzup???- Monitors server status- Monitors host resource usage

- WhatsUp Gold- Network monitoring- Fault detection

– Security Service component- Various freeware or public domain packages- Monitor and evaluate security and report status

288625-EMD-002, Rev. 02


• Management CSCI (MCI) (Cont.)– Accountability Management Service (AMS) component

- Custom software- Account Management Tool (for User Registration and User Profile

updates)- Order Tracking Tool- Sybase ASE Server/Sybase Replication Server

289625-EMD-002, Rev. 02


• Management CSCI (MCI) (Cont.)– Trouble Ticketing component

- Custom-configured COTS software: Remedy Action Request System

– Network Backup/Restore component- COTS software: Legato Networker

– ASTER E-Mail Header Handler component- Custom scripts work with COTS e-mail to add a formatted header

to all e-mail exchanges between the ASTER Ground Data System and ECS

290625-EMD-002, Rev. 02

O

MSS

MCI - Accountability

EcMsAcDAACRegUserGUI

ASE(Sybase)

Order TrackingDatabase

User ProfileDatabase

EcMsAcRegUserSrvr

EcMsAcOrderSrvr

EcMsAcOrderGUI

EcMsAcSMCRegUserGUI

SMCSybaseReplicationServer

ASE(Sybase)

UserProfileDatabase

EcMsAcRegUserSrvr

SybaseReplicationServer

(At SMC)

Subsystems and CSCIs: MSS (Cont.)AMS Architecture and Interfaces

291625-EMD-002, Rev. 02

O

MSS



ASE(Sybase)



EcMsAcRegUserSrvr

EcMsAcOrderSrvr

EcMsAcOrderGUI

EcMsAcSMCRegUserGUI


ASE(Sybase)

UserProfileDatabase

EcMsAcRegUserSrvr


(At SMC)


Request user registration/User

Profile update UserProfiledata

292625-EMD-002, Rev. 02

O

MSS



ASE(Sybase)



EcMsAcRegUserSrvr

EcMsAcOrderSrvr

EcMsAcOrderGUI

EcMsAcSMCRegUserGUI


ASE(Sybase)

UserProfileDatabase

EcMsAcRegUserSrvr


(At SMC)


UserProfile request informa-tion

Insert/UpdateUser Profile information

293625-EMD-002, Rev. 02

O

MSS



ASE(Sybase)



EcMsAcRegUserSrvr

EcMsAcOrderSrvr

EcMsAcOrderGUI

EcMsAcSMCRegUserGUI


ASE(Sybase)

UserProfileDatabase

EcMsAcRegUserSrvr


(At SMC)


Create/update/view User Profiles(DAAC operator logged in remotely)

UserProfile informa-tion

294625-EMD-002, Rev. 02

O

MSS



ASE(Sybase)



EcMsAcRegUserSrvr

EcMsAcOrderSrvr

EcMsAcOrderGUI

EcMsAcSMCRegUserGUI


ASE(Sybase)

UserProfileDatabase

EcMsAcRegUserSrvr


(At SMC)


User Profile database replication

295625-EMD-002, Rev. 02

O

MSS



ASE(Sybase)



EcMsAcRegUserSrvr

EcMsAcOrderSrvr

EcMsAcOrderGUI

EcMsAcSMCRegUserGUI


ASE(Sybase)

UserProfileDatabase

EcMsAcRegUserSrvr


(At SMC)


User Profile information

User Profile request

296625-EMD-002, Rev. 02

O

MSS



ASE(Sybase)



EcMsAcRegUserSrvr

EcMsAcOrderSrvr

EcMsAcOrderGUI

EcMsAcSMCRegUserGUI


ASE(Sybase)

UserProfileDatabase

EcMsAcRegUserSrvr


(At SMC)


User Profiles

Request User Profiles (read only)

297625-EMD-002, Rev. 02

O

MSS



ASE(Sybase)



EcMsAcRegUserSrvr

EcMsAcOrderSrvr

EcMsAcOrderGUI

EcMsAcSMCRegUserGUI


ASE(Sybase)

UserProfileDatabase

EcMsAcRegUserSrvr


(At SMC)


Order/request tracking information

Request order; request order

status

Insert/update order; request

order status

298625-EMD-002, Rev. 02

O

MSS



ASE(Sybase)



EcMsAcRegUserSrvr

EcMsAcOrderSrvr

EcMsAcOrderGUI

EcMsAcSMCRegUserGUI


ASE(Sybase)

UserProfileDatabase

EcMsAcRegUserSrvr


(At SMC)


Update order status

Update order status

299625-EMD-002, Rev. 02

O

MSS



ASE(Sybase)



EcMsAcRegUserSrvr

EcMsAcOrderSrvr

EcMsAcOrderGUI

EcMsAcSMCRegUserGUI


ASE(Sybase)

UserProfileDatabase

EcMsAcRegUserSrvr


(At SMC)


Order tracking

information;

request communications

support

Request order status (MTM Gwy); configuration parameters, media

options (Registry)

300625-EMD-002, Rev. 02

O

MSS



ASE(Sybase)



EcMsAcRegUserSrvr

EcMsAcOrderSrvr

EcMsAcOrderGUI

EcMsAcSMCRegUserGUI


ASE(Sybase)

UserProfileDatabase

EcMsAcRegUserSrvr


(At SMC)


Create order(ODFRM)

301625-EMD-002, Rev. 02

O

MSS



ASE(Sybase)



EcMsAcRegUserSrvr

EcMsAcOrderSrvr

EcMsAcOrderGUI

EcMsAcSMCRegUserGUI


ASE(Sybase)

UserProfileDatabase

EcMsAcRegUserSrvr


(At SMC)


Order/request status/ information

Update order; request order

status/information

302625-EMD-002, Rev. 02

O

MSS



ASE(Sybase)



EcMsAcRegUserSrvr

EcMsAcOrderSrvr

EcMsAcOrderGUI

EcMsAcSMCRegUserGUI


ASE(Sybase)

UserProfileDatabase

EcMsAcRegUserSrvr


(At SMC)


Management information (e.g., User Profile data, order information)

Request management services

303625-EMD-002, Rev. 02


• Management Logistics CSCI (MLCI)– Implements Configuration Management services– Inventory/Logistics/Maintenance (ILM) Manager component

- Customized COTS software: a Remedy application- Tracks and maintains key data on ECS contract-purchased

equipment, hardware, COTS software, COTS documentation (hardware and software), spares and consumable items, and Government Furnished Equipment (GFE)

- Stores and maintains detailed maintenance data on hardware to the component level, including corrective maintenance

304625-EMD-002, Rev. 02


• Management Logistics CSCI (MLCI) (Cont.)– Software Change Manager component

- Consists of COTS and custom software- ClearCase (with some customization)- Supporting UNIX scripts

- Helps organize and partition software, control software changes and versions, and assemble sets of software for release

305625-EMD-002, Rev. 02


• Management Logistics CSCI (MLCI) (Cont.)– Software License Manager component

- COTS software- FLEXlm server daemon (lmgrd) with its associated command

line utilities- FLEXlm-enabled COTS Application - client software within

vendor products communicates with FLEXlm’s license server and vendor daemons to request licenses for product users to run

- Manages network licensing activities associated with using COTS products; maintains information about license provisions, metersuse of installed licenses, and reports on licensing events and statistics

306625-EMD-002, Rev. 02


DataData Pool

(DPL)DATAPOOL

ElectronicProducts


(DPS) Data Proc

0110001011001011001011010111010110001011

InventorySearch


Archive


SDSRVDDIST

STMGT

Inven-toryDB

Collection

Insertand

Retrieve


PUSH


PlanningSubsystem

(PLS) Planning

abc

def

ghi

jkl

mno

PULLSystem Mgmt

SubsystemMSS

AcquireRequests

ProductOrders



Meta-data

ProductRequests

EMOS


Searchand

Access


Data MgmtSubsystem

(DMS)


Searchand

Download





Data

DataIngest



MSS


ManagementLogistics CSCI

(MLCI)

Inventory/Logistics/Maintenance Mgr

Remedy ARS (COTS)

SoftwareChange MgrClearCase (COTS)

SoftwareLicense Mgr

lmgrd (FLEXlm server daemon) (COTS)FLEXlm-enabled COTS Application

ManagementCSCI (MCI)

Network andEnterprise

ManagementWhatsUp GoldWhazzup???

Securityanlpasswd (COTS)

TCP Wrappers (COTS)Tripwire (COTS)Crack (COTS)

AccountabilityEcMsAcRegUserSrvr

EcMsAcSMCRegUserGUIEcMsAcDAACRegUserGUI

EcMsAcOrderSrvrEcMsAcOrderGUI

Sybase Replication ServerSybase ASE

Trouble Ticketaruser GUI (COTS)

aradmin GUI (COTS)notifier GUI (COTS)

notifier daemon (COTS)Remedy ARS (COTS)

armail daemon (COTS)Sendmail Advanced Message

Server (SAMS) (COTS)Sybase ASE (COTS)

NetworkBackup/RestoreNetworker Server (COTS)Networker Client (COTS)

ASTER E-mailHeader Handler

UNIX sendmail daemon (COTS)EcMsAsAddHeader.pl

EcMsAsRemoveHeader.ple-mail utility (COTS)

307625-EMD-002, Rev. 02


DataData Pool

(DPL)DATAPOOL

ElectronicProducts


(DPS) Data Proc

0110001011001011001011010111010110001011

InventorySearch


Archive


SDSRVDDIST

STMGT

Inven-toryDB

Collection

Insertand

Retrieve

PUSH


PlanningSubsystem

(PLS) Planning

abc

def

ghi

jkl

mno

PULLSystem Mgmt

SubsystemMSS

AcquireRequests

ProductOrders



Meta-data

ProductRequests

EMOS


Searchand

Access


Data MgmtSubsystem

(DMS)


Searchand

Download





Data

DataIngest


Subsystems and CSCIs: CSS

CSS/DistributedCommunications Software

308625-EMD-002, Rev. 02

Subsystems and CSCIs: CSS (Cont.)

• Communications Subsystem (CSS)– Provides for interconnection of users and

service providers and transfer of information within ECS and between ECS and other EOSDIS components, including a machine-to-machine gateway for SIPS access to ECS data

– Supports and interacts with the System Management Subsystem (MSS), ECS Mission Operations Segment (EMOS), and all other subsystems

– Uses several COTS tools: RogueWave class libraries, Builder Xcessory (GUI Builder tool), Sybase ASE Server (for Subscription Server insert, search, and update), UNIX Network Services


309625-EMD-002, Rev. 02


• Distributed Computing Configuration Item (DCCI)– Subscription Server (SBSRV) and GUI components

- Detects previously defined events- Performs specified actions for clients that have previously

subscribed to those events (e.g., science granule insertion, metadata update, science granule deletion)

- Being replaced by Spatial Subscription Server (NBSRV)– ASTER DAR Gateway Server (hosted at the LP DAAC)

- Provides interoperability between ASTER DAR Client GUI tool and the DAR API which interfaces to the ASTER servers

– ASTER E-Mail Parser Gateway Server- Support for automated delivery of ASTER Expedited Data Sets

(EDS) from ECS to ASTER Ground Data System (GDS)– Message-Oriented Java Object (MOJO) Gateway Server

- Gateway for access by the ASTER DAR Tool to all ECS Services; directs DARs to GDS via ASTER DAR Gateway

310625-EMD-002, Rev. 02


• Distributed Computing Configuration Item (DCCI) (Cont.)– Configuration Registry Server

- Provides a single interface to retrieve configuration attribute-value pairs for ECS servers from the Configuration Registry Database, via Sybase Server

- Provides an interface to retrieve distribution options for ECS Servers from the Configuration Registry Database

– Machine-to-Machine Gateway Server- Provides an automated search and order capability to allow the

Science Investigator-Led Processing Systems (SIPS) to reprocess data externally from the ECS

– CCS Middleware Support Group- CCS Name Server enables clients to locate and communicate with

the various ECS servers

311625-EMD-002, Rev. 02


• Distributed Computing Configuration Item (DCCI) (Cont.)– File Transfer Protocol (FTP) component (Remote File Access

Group)- Standard application for file transfers (standard UNIX utility with

CSS wrapper classes applied)– FTP Notification component (Remote File Access Group)

- Provides notification of successful FTP pulls from a pull area– Bulk Data Server (BDS) component (Remote File Access

Group)- COTS product (BDSpro) that permits fast file transfer over high-

speed networks (such as Gigabit Ethernet)– Network File System (NFS) component (Remote File Access

Group)- COTS product that provides a file-sharing system among

computers

312625-EMD-002, Rev. 02


• Distributed Computing Configuration Item (DCCI) (Cont.)– Filecopy component (Remote File Access Group)

- A simple utility to copy large files from a specified source to a specifed destination, with compression options

– Mail Support Group- Custom and COTS software that provides electronic mail, with an

interactive interface and an object-oriented application program interface

– Virtual Terminal component- COTS software (Telnet) that provides ECS Operations personnel

with the capability for remote logon from one ECS machine to another

313625-EMD-002, Rev. 02


• Distributed Computing Configuration Item (DCCI) (Cont.)– Cryptographic Management Interface (CMI)

- CMI classes provide the requesting process with a server accountand a randomly generated password so the server can access security required services (i.e., Sybase ASE)

- Passwords (and optionally login names) are generated dynamicallybased on a psuedo-random number used as the seed for the password

– Domain Name Server (DNS)- COTS product that provides information about host names and

addresses on a network by querying and answering queries- Performs naming between hosts within the local administrative

domain and across domain boundaries- Note: The external DNS is located on the Firewall in ISS

314625-EMD-002, Rev. 02


• Distributed Computing Configuration Item (DCCI) (Cont.)– Infrastructure Libraries Group

- Provides a set of services to facilitate the implementation of client-server applications; includes …- Process Framework (PF)- Service Request Framework (SRF)- Message Passing- Universal References (URs)- Event Logging- Server Locator- Failure Recovery Framework- EcPo Connections- Time Service

315625-EMD-002, Rev. 02


DataData Pool

(DPL)DATAPOOL

ElectronicProducts


(DPS) Data Proc

0110001011001011001011010111010110001011

InventorySearch


Archive


SDSRVDDIST

STMGT

Inven-toryDB

Collection

Insertand

Retrieve

PUSH


PlanningSubsystem

(PLS) Planning

abc

def

ghi

jkl

mno

PULLSystem Mgmt

SubsystemMSS

AcquireRequests

ProductOrders



Meta-data

ProductRequests

EMOS


Searchand

Access


Data MgmtSubsystem

(DMS)


Searchand

Download





Data

DataIngest



DomainName Server

resolver

Mail SupportE-Mail Server

Distributed ComputingConfiguration Item (DCCI)

ASTER DARGateway

EcGwDARServer

E-Mail ParserGateway

EcCsEmailParser

FTPNotification

CsFtFTPNotify

Bulk DataServer

BDSpro

NetworkFile System

NFS

CCSMiddleware

SupportEcCsIdNameServer

SubscriptionServer

EcSbSubServerEcSbGui

File TransferProtocol

FTP

FilecopyEcUtFileCopyEcUtFileExecVirtual

TerminalVT

CryptographicManagement

InterfaceCMI

InfrastructureLibrariesPF, SRF, URs,

Message Passing,Event Logging,Server Locator,

Failure RecoveryFramework,

EcPo Connections,Time Service

MOJOGateway

EcCsMojoGateway

ConfigurationRegistry Srvr.EcCsRegistryServer

Machine-to-Machine Gwy.

EcCsMtMGateway


316625-EMD-002, Rev. 02

ECS Operational FunctioningIN

SERT

ION

RETRIEVAL

PRODUCTION


Store


Deliver

Generate Plan

Ingest

CAUTIONSome of the following pages display numbers that may

indicate the sequence in which graphic elements appear in the dynamic presentations. These numbers do not

reliably indicate the sequence of events in ECS operation.For more complete event sequence information, consult

Document 313-EMD-001.

317625-EMD-002, Rev. 02

ECS Release 7 Focus

TerraMETEOR - SAGE III

DAOAqua

Aura

318625-EMD-002, Rev. 02

ASTER Scenario

ASTER Goals- ASTER DAR Tool Usage- On-Demand Processing and Chaining- SCF QA Metadata Update Workaround- Simplified ASTER Expedited Data Support- Data Tape Ingest

ASTER PreconditionsASTER ESDTs Inserted into ECS-AST_ANC, AST_EXP, AST_L1A, AST_L1BT,AST_09T, AST_04, AST_05, AST_08, GDAS0ZFH

ASTER PGEs passed SSI&T and installed- ACT, ETS, BTS

Ancillary data inserted into Data Server

319625-EMD-002, Rev. 02

ASTER Scenario: DAR SupportIN

SERT

ION

RETRIEVAL

PRODUCTION


Store


Deliver

Generate Plan

Ingest

DAR SubmissionData SubscriptionOn-Demand Request

320625-EMD-002, Rev. 02

RETRIEVAL

SubscribeRegister

Subscription

Deliver

ASTER Scientist decidesto request ASTER datarequiring a Data AcquisitionRequest

Search& Order

OrderProducts

Store

DAR Support

321625-EMD-002, Rev. 02

ASTER: Client Request Process

ASTER Scientist determines an area of interest. The scientistdecides to request an ASTER data take over that area, using theASTER DAR Tool.

1. Request DataAcquisition

2. SubmitDAR


SDSRVDDIST

STMGT

Inven-toryDB

Collection

CommunicationsSubsystemMJGWY

Data MgmtSubsystem

DDICT



abc

def

ghi

jkl

mno


0110001011001011001011010111010110001011

Client

Systems MgmtSubsystem

(MSS)


(OMS)


322625-EMD-002, Rev. 02


(MSS)















NetworkBackup/Restore

Networker Server (COTS)Networker Client (COTS)





(MLCI)


Remedy ARS (COTS)


SoftwareLicense Mgr



DomainName Server

resolver



ASTER DARGateway

EcGwDARServer


EcCsEmailParser

FTPNotificationCsFtFTPNotify

Bulk DataServerBDSpro

NetworkFile System

NFS

CCSMiddleware


SubscriptionServer



FTP

FilecopyEcUtFileCopyEcUtFileExec

VirtualTerminal

VT


InterfaceCMI

InfrastructureLibraries

PF, SRF, URs,Message Passing,

Event Logging,Server Locator,



MOJOGateway

EcCsMojoGateway


Machine-to-Machine Gwy.EcCsMtMGateway

ASTER: CSCI/Component Rolein Client Request



(ODFRM) CSCI

HTML UserInterface

ODFRM

CGIPrograms

EcClOdRequest

Desktop(DESKT) CSCI

User ProfileGateway




EOSView

ASTER DARTool

EcClWbJdt

Provides GUI to specify parameters for data take; submits request to ASTER GDS through CSS

5

Server; provides UserProfile Information andauthenticates privileges

2

Holds User Profile

information,

including special

privileges1

Submits user information

to CCS Middleware;

provides access to

ECS services from the

Java front end of ASTER

DAR Tool3

Provides session ID

to Mojo Gateway

4

323625-EMD-002, Rev. 02

RETRIEVAL

SubscribeRegister

Subscription

Deliver

ECS submits DAR to ASTER GroundData System (GDS) in Japanreceiving a DARid in return

Store

DAR Support (Cont.)

GDS

Search& Order

OrderProducts

324625-EMD-002, Rev. 02

ASTER: Request Data Take Process

ASTER DAR Gateway submits a request for a data take over the areaof interest. GDS responds with a DARid.


SDSRVDDIST

STMGT

Inven-toryDB

Collection

CommunicationsSubsystemDAR GW



abc

def

ghi

jkl

mno


0110001011001011001011010111010110001011

Client

GDS1. Request Data Take

Data MgmtSubsystem

DDICT


(MSS)


(OMS)


325625-EMD-002, Rev. 02


DomainName Server

resolver



ASTER DARGateway

EcGwDARServer


EcCsEmailParser

FTPNotification

CsFtFTPNotify

Bulk DataServer

BDSpro

NetworkFile System

NFS

CCSMiddleware


SubscriptionServer



FTP


TerminalVT


InterfaceCMI





MOJOGateway

EcCsMojoGateway



EcCsMtMGateway

ASTER: CSCI/Component Rolein Data Take Request

Submits request to ASTER GDS; receives DARid from GDS

1

Provides access byASTER DAR Tool toECS Services

2

326625-EMD-002, Rev. 02

RETRIEVAL

Deliver

Subscription is submitted onbehalf of user for notificationon receipt of AST_L1BT (TIR)data

Store

DAR Support (Cont.)

Search& Order

OrderProducts

Subscribe

RegisterSubscription

MoJoGateway

327625-EMD-002, Rev. 02

ASTER: Submit Subscription Process

MoJo Gateway submits subscription for notification on the occurrence ofAST_L1BT:Insert event, qualified with the DARid.

1. Subscribe (L1BT:Insert, e-mailnotification)


SDSRVDDIST

STMGT

Inven-toryDB

Collection

CommunicationsSubsystem

SBSRV



abc

def

ghi

jkl

mno


0110001011001011001011010111010110001011

Client


(MSS)

Data MgmtSubsystem

DDICT


(OMS)


328625-EMD-002, Rev. 02


DomainName Server

resolver



ASTER DARGateway

EcGwDARServer


EcCsEmailParser

FTPNotification

CsFtFTPNotify

Bulk DataServer

BDSpro

NetworkFile System

NFS

CCSMiddleware


SubscriptionServer



FTP


TerminalVT


InterfaceCMI





MOJOGateway

EcCsMojoGateway



EcCsMtMGateway

ASTER: CSCI/Component Rolein Subscription Submission

Registers subscription and assignsit to persistent storage

2

Submits subscription(Subscribe toL1BT:Insert)

1

329625-EMD-002, Rev. 02

RETRIEVAL

SubscribeRegister

Subscription

Deliver

ASTER Scientist decidesto request production ofa higher level ASTERproduct from the data thatare to be collected

Search& Order

OrderProducts

Store

DAR Support (Cont.)

330625-EMD-002, Rev. 02

ASTER: On-Demand Data Processing Request Process

ASTER Scientist wants AST_08 (L2 Surface Temperature product) based onthe AST_L1BT (TIR - Thermal InfraRed - product) resulting from GDSinitial processing of the data collected for the DAR, and uses the EOS DataGateway (EDG) Web Client to submit the request.


SDSRVDDIST

STMGT

Inven-toryDB

Collection


SBSRV



0110001011001011001011010111010110001011

Client


(MSS)


abc

def

ghi

jkl

mno 2. Submit selections (e.g., input,output, distribution)

3. Create order-trackingelements in the MSSorder-tracking database

EDG

UserAuthentication

4. Submit on-demandrequest

1. Request higher-level data product

Data MgmtSubsystem

DDICT


(OMS)


331625-EMD-002, Rev. 02


DataStore





EcPlPrEditor_IF


EcPlWbEcPlTl



EcPlOdMgr



EcPlProdStrat


EcPlPRGenerator

PlanningSubsystem

(PLS) Planning

abcdefghijklmno


(MSS)





















(MLCI)


Remedy ARS (COTS)


SoftwareLicense Mgr




(ODFRM) CSCI

HTML UserInterface

ODFRM

CGIPrograms

EcClOdRequest

Desktop(DESKT) CSCI

User ProfileGateway




EOSView

ASTER DARTool

EcClWbJdt

ASTER: CSCI/Component Role in On-Demand Request

Holds User Profileinformation, including specialprivileges

1

Data MgmtSubsystem

(DMS)



EcDmMaintTool




EcDmDictService


V0 to ECSGateway Server

EcDmV0ToECSGateway

Records order-tracking elements in the Order-Tracking Database

4

Provides User Profile

information;

authenticates user

privileges

2

Receives user selections for AST_08 from EDG; submits request to MSS to create astanding order for FTPpush of AST_08; passes theon-demand order to PLS with OrderID and RequestID

3

ODPRM receives standing order (FTPpush of AST_08) and registers subscription for notification upon arrival of L1B product

5

332625-EMD-002, Rev. 02

ASTER Scenario: Chaining and On-Demand Production

INSE

RTIO

NRETRIEVAL

PRODUCTION


Store


Deliver

Generate Plan

Ingest

Data InsertionData NotificationOn-demand ProductionStanding Order DeliveryQA Update

333625-EMD-002, Rev. 02

INSE

RTIO

N

Subscribe

Notification

Ingest

Media

StoreArchive

Data

CatalogueData

Some time later, after receivingSony DTF-2 tape from GDS,operator mounts tape and beginsingest of data

Chaining and On-Demand Production (Cont.)

SonyDTF-2Tape

334625-EMD-002, Rev. 02

ASTER: DTF-2 Tape Ingest Process

After receiving Sony DTF-2 tape in a shipment, DAAC Operatormounts tape and begins ingest activities.Tape contains AST_L1BT (L1B TIR) data.


SDSRVDDIST

STMGT

Inven-toryDB

Collection


SBSRV


abc

def

ghi

jkl

mno


0110001011001011001011010111010110001011

Client

IngestSubsystem Ingest1. Mount tape

4. Get mcf (AST_L1BT)

Data MgmtSubsystem

DDICT


(OMS)


2. Read PDR file3. Read AST_L1BT

files

SonyDTF-2Tape

335625-EMD-002, Rev. 02

IngestSubsystem

(INS) Ingest

Ingest(INGST) CSCI


ProcessEcInPolling


ProcessEcInGUI

EcInTapeReaderGUI


ProcessEcInReqMgr


ProcessEcInGran

IngestDatabase



EcInEmailGWServer


SDSRVDDIST

STMGT

Inven-toryDB

Collection


Cache/StagingManager Process

EcDsStCacheManagerServerEcDsStStagingDiskServer

MediaServer Process


EcDsStCopyServer

Pull MonitorProcess



EcDsStmgtGui




ScienceData Server





Granule DeletionAdministration Tool

EcDsGranuleDelete



EcDsSCLI


DataDistribution



EcDsDdistGui



EcDsDdDCLI

ASTER: CSCI/Component Rolein Ingest DTF-2 Tape Operations

Packages Ingest Request into Granules

3

OperatorInterface; reads delivery record file and data files from DTF-2 Tape

1

Staging Disk Server

receives request from

Ingest to create staging

disk and reserves space

6Provides metadata configuration file (mcf)

5

Detects PDR; creates ingest request

2

Requests metadata configuration file (mcf) from SDSRV; requests staging disk4

Request ManagerEcDsStRequestManagerServer

336625-EMD-002, Rev. 02

INSE

RTIO

N

Subscribe

Notification

StoreArchive

Data

CatalogueData

Ingested AST_L1BTdata are archived


Ingest

Media

337625-EMD-002, Rev. 02

ASTER: Ingest Archive Insertion Process

Archive AST_L1BT (L1B TIR) data granules.


SDSRVDDIST

STMGT

Inven-toryDB

Collection


SBSRV


abc

def

ghi

jkl

mno


0110001011001011001011010111010110001011

Client


1. Insert (AST_L1BT)

2. Insert (AST_L1BTfiles)

Data MgmtSubsystem

DDICT


(OMS)


338625-EMD-002, Rev. 02

IngestSubsystem

(INS) Ingest

Ingest(INGST) CSCI


ProcessEcInPolling


ProcessEcInGUI

EcInTapeReaderGUI


ProcessEcInReqMgr


ProcessEcInGran

IngestDatabase



EcInEmailGWServer


SDSRVDDIST

STMGT

Inven-toryDB

Collection




MediaServer Process


EcDsStCopyServer

Pull MonitorProcess



EcDsStmgtGui




ScienceData Server






EcDsGranuleDelete



EcDsSCLI


DataDistribution



EcDsDdistGui



EcDsDdDCLI

ASTER: CSCI/Component Rolein Ingest Archive Insertion

Builds metadata

file; requests

SDSRV to

validate

metadata;

requests

SDSRV to insert

received files

1

Validates metadata; requests archiving of granule files

2


Reads inserted files from Ingest staging disks

3

339625-EMD-002, Rev. 02

INSE

RTIO

N

Subscribe

Notification

StoreArchive

Data

CatalogueData

Update catalogue with referenceto new L1BT data

Ingest

Media


340625-EMD-002, Rev. 02

ASTER: Inventory (Metadata) Update Process

Add metadata for AST_L1BT (L1B TIR) data granulesto the Sybase/SQS database.


SDSRVDDIST

STMGT

Inven-toryDB

Collection


SBSRV


abc

def

ghi

jkl

mno


0110001011001011001011010111010110001011

Client


1. Add AST_L1BT metadata

Data MgmtSubsystem

DDICT


(OMS)


341625-EMD-002, Rev. 02


SDSRVDDIST

STMGT

Inven-toryDB

Collection


Cache/StagingManager ProcessEcDsStCacheManagerServer


MediaServer Process


EcDsStCopyServer

Pull MonitorProcess



EcDsStmgtGui




ScienceData Server






EcDsGranuleDelete



EcDsSCLI


DataDistribution



EcDsDdistGui



EcDsDdDCLI

ASTER: CSCI/Component Rolein Inventory (Metadata) Update

Adds validated metadata to the inventory

2

Sends validated metadata to database 1

342625-EMD-002, Rev. 02

PRODUCTION

Subscribe

NotificationRegister

Subscription

Store

ArchiveData

LocateData

CatalogueData

RetrieveData

ProcessOn

Demand

Insert terminates with an insert event notification to Subscribe. Subscribee-mails ASTER Scientist notice of a completed AST_L1BT granule insert, and also notifies Planning


343625-EMD-002, Rev. 02

ASTER: Event Notification Process

Notify all AST_L1BT:Insert event subscribers whoseDARid numbers are matched with the ingested granules.


SBSRV


abc

def

ghi

jkl

mno


0110001011001011001011010111010110001011

Client


1. Trigger(AST_L1BT:Insert,XARids)


SDSRVDDIST

STMGT

Inven-toryDB

Collection

2. E-mail notification3. Notify (AST_L1BT:Insert,

direct)

Data MgmtSubsystem

DDICT


(OMS)


344625-EMD-002, Rev. 02


DomainName Server

resolver



ASTER DARGateway

EcGwDARServer


EcCsEmailParser



NetworkFile System

NFS

CCSMiddleware


SubscriptionServer



FTP


VirtualTerminal

VT


InterfaceCMI






MOJOGateway

EcCsMojoGateway




DataStore





EcPlPrEditor_IF


EcPlWbEcPlTl



EcPlOdMgr



EcPlProdStrat


EcPlPRGenerator

PlanningSubsystem

(PLS) Planning

abcdefghijklmno

ASTER: CSCI/Component Rolein Event Notification

Sendmail deliverse-mail message for user to mailhost

3

Receives notice of AST_L1BT:Insert4


SDSRVDDIST

STMGT

Inven-toryDB

Collection




MediaServer Process


EcDsStCopyServer

Pull MonitorProcess



EcDsStmgtGui




ScienceData Server






EcDsGranuleDelete



EcDsSCLI


DataDistribution



EcDsDdistGui



EcDsDdDCLI

Triggers AST_L1BT:Insert event; notifies the correct subscription server

1

Fires subscriptions; sends notifications of the AST_L1BT:Insert event

2

345625-EMD-002, Rev. 02


PRODUCTION

ProcessOn

Demand

Subscribe


Subscription

Store

ArchiveData

LocateData

CatalogueData

RetrieveData

Planning recognizes the need to run ACT PGE before ETS. Creates data processing requests (DPRs) for ACT and ETS, and registers subscriptions for input products that are not available in the archive.

346625-EMD-002, Rev. 02

ASTER: PGE Chaining

Input: 1 AST_L1BT

Requested Output: 1 AST_08

ETS (PGE)

1 AST_05

1 AST_ANC1 AST_09T

ACT (PGE)

1 GDAS0ZFH 24 AST_ANC1 AST_L1BT

1 AST_08

347625-EMD-002, Rev. 02

ASTER: Sequenced Production Request ProcessPlanning recognizes that, in order to run ETS algorithm on AST_L1BT (L1B TIR),ACT algorithm must be run first. Planning creates DPRs for ACT and ETS, with the AST_09T (L2 Surface Radiance TIR) output feeding the ETSalgorithm, submitting subscriptions for data not available in the archive.


SDSRVDDIST

STMGT

Inven-toryDB

Collection



abc

def

ghi

jkl

mno


0110001011001011001011010111010110001011

Client


SBSRV1. Submitsubscriptions

2. Submit searchfor inputgranules

3. Search database

4. Return results

Data MgmtSubsystem

DDICT


(OMS)


348625-EMD-002, Rev. 02


DomainName Server

resolver



ASTER DARGateway

EcGwDARServer


EcCsEmailParser



NetworkFile System

NFS

CCSMiddleware


SubscriptionServer



FTP


VirtualTerminal

VT


InterfaceCMI






MOJOGateway

EcCsMojoGateway




SDSRVDDIST

STMGT

Inven-toryDB

Collection




MediaServer Process


EcDsStCopyServer

Pull MonitorProcess



EcDsStmgtGui




ScienceData Server






EcDsGranuleDelete



EcDsSCLI


DataDistribution



EcDsDdistGui



EcDsDdDCLI


DataStore





EcPlPrEditor_IF


EcPlWbEcPlTl



EcPlOdMgr



EcPlProdStrat


EcPlPRGenerator

PlanningSubsystem

(PLS) Planning

abcdefghijklmno

ASTER: CSCI/Component Rolein Sequenced Production Request

Stores subscriptionsin the Sybase Database

2

Stores DataProcessing Requests5 Breaks down

Query and

generates calls

to inventory

database;

packages and

returns results

Database; contains metadata for search and retrieval

3

4

ODPRM creates DPRs;submits subscriptions;searches SDSRV for input granules

1

625-EMD-000 Revised 01/10/2005

349625-EMD-002, Rev. 02


PRODUCTION

Subscribe


Subscription

Store

ArchiveData

LocateData

CatalogueData

RetrieveData

Planning releases the On-Demand jobs, including the DPR for ACT and a dependent one for ETS

ProcessOn

Demand

350625-EMD-002, Rev. 02

ASTER: Job Activation Process

Planning releases the On-Demand jobs; the release activates the ACT DPR forprocessing.


SDSRVDDIST

STMGT

Inven-toryDB

Collection



0110001011001011001011010111010110001011

Client


SBSRV


abc

def

ghi

jkl

mno

1. On-Demand jobsrelease

Data MgmtSubsystem

DDICT


(OMS)


351625-EMD-002, Rev. 02

Data ProcessingSubsystem(DPS) Data Proc

0110001011001011001011010111010110001011






EcDpAtOpDbGui



Check SoftwareTools



Insert/AcquireTools

EcDpAtINSTALLEcDpAtInsertStaticFile





for SSI&T)


ExecutionManagement


PGEManagement


DeletionServer

EcDpPrDeletion



AutoSysCOTS Product

QAMonitor

EcDpPrQaMonitorGUI

DataStore


Ground EventProcessEcDpPrGE







DataStore





EcPlPrEditor_IF


EcPlWbEcPlTl



EcPlOdMgr



EcPlProdStrat


EcPlPRGenerator

PlanningSubsystem

(PLS) Planning

abcdefghijklmno

ASTER: CSCI/Component Rolein Job Activation

Releases the DPR for ACTto AutoSys for jobprocessing 2

Adds UR for new granule (AST_L1BT)to data context; matches metadata against job input needs; notifies Job Management to release ACT

1

352625-EMD-002, Rev. 02


PRODUCTION

Subscribe


Subscription

Store

ArchiveData

LocateData

CatalogueData

RetrieveData

Job locates AST_L1BT, AST_ANC (ASTER ancillary data set), and GDAS0ZFH (NCEP ancillary) data required for ACT algorithm

ProcessOn

Demand

353625-EMD-002, Rev. 02

ASTER: Input Data Location Process

ACT locates required AST_L1BT (L1B TIR), AST_ANC (ASTER ancillary dataset), and GDAS0ZFH (NCEP ancillary) input data granules.


SDSRVDDIST

STMGT

Inven-toryDB

Collection


Client


SBSRV

1. Search (AST_L1BT, AST_ANC, &GDAS0ZFH)

2. Query Sybase


abc

def

ghi

jkl

mno


0110001011001011001011010111010110001011

Data MgmtSubsystem

DDICT


(OMS)


354625-EMD-002, Rev. 02


0110001011001011001011010111010110001011






EcDpAtOpDbGui



Check SoftwareTools



Insert/AcquireTools






for SSI&T)


ExecutionManagement


PGEManagement


DeletionServer

EcDpPrDeletion



AutoSysCOTS Product

QAMonitor

EcDpPrQaMonitorGUI

DataStore









SDSRVDDIST

STMGT

Inven-toryDB

Collection




MediaServer Process


EcDsStCopyServer

Pull MonitorProcess



EcDsStmgtGui




ScienceData Server






EcDsGranuleDelete



EcDsSCLI


DataDistribution



EcDsDdistGui



EcDsDdDCLI

ASTER: CSCI/Component Rolein Input Data Location

Releases and sequencesDPRs (Jobs)

1

Controls staging andprocessing activities 2


Retrieves metadata for requested input granules from the Sybase/SQS database

3

4

355625-EMD-002, Rev. 02


PRODUCTION

Subscribe


Subscription

Store

ArchiveData

LocateData

CatalogueData

RetrieveData

Retrieve AST_L1B, AST_ANC (ASTER ancillary data set), and GDAS0ZFH granules as input to ACT; PGE execution begins

ProcessOn

Demand

356625-EMD-002, Rev. 02

ASTER: Job Staging Process

ACT production job retrieves required AST_L1BT (L1B TIR), AST_ANC (ASTER ancillary data set), and GDAS0ZFH (NCEP ancillary) input datagranules.


SDSRVDDIST

STMGT

Inven-toryDB

Collection


Client


SBSRV


abc

def

ghi

jkl

mno

1. Acquire (AST_L1BT,AST_ANC, &GDAS0ZFH, ftpPush)


0110001011001011001011010111010110001011

2. Distribution Requests (AST_L1BT, AST_ANC, & GDAS0ZFH files, ftpPush)

3. Copy (AST_L1BT, AST_ANC, &GDAS0ZFH files, ftpPush)

4. ftpPush (AST_L1BT,AST_ANC, &GDAS0ZFH files)

Data MgmtSubsystem

DDICT


(OMS)


357625-EMD-002, Rev. 02


0110001011001011001011010111010110001011






EcDpAtOpDbGui



Check SoftwareTools



Insert/AcquireTools






for SSI&T)


ExecutionManagement


PGEManagement


DeletionServer

EcDpPrDeletion



AutoSysCOTS Product

QAMonitor

EcDpPrQaMonitorGUI

DataStore









SDSRVDDIST

STMGT

Inven-toryDB

Collection




MediaServer Process


EcDsStCopyServer

Pull MonitorProcess



EcDsStmgtGui




ScienceData Server






EcDsGranuleDelete



EcDsSCLI


DataDistribution



EcDsDdistGui



EcDsDdDCLI

ASTER: CSCI/Component Rolein Job Staging

Allocates space in response to Distribution Requests 6

Releases and

sequencesDPRs (Jobs)

1

Requests staging disks for granule files; requests granule file retrieval by STMGT

5

Stages granule files to read-only cache disks

7

Controls staging and processing activities; requests MCF; handles requests to the Archive for retrieving granules

2


Provides Resource

Manager for ftpPushes

8Database; contains metadata for search and retrieval

Provides Metadata Configuration File (MCF); retrieves metadata; requests metadata Staging Disks; creates and passes distribution requests and metadata files to DDIST

4

3

358625-EMD-002, Rev. 02


PRODUCTION

Subscribe


Subscription

Store

ArchiveData

LocateData

CatalogueData

RetrieveData

Archive newly created AST_09T (L2 Surface Radiance TIR) granule after completion of ACT PGE;

ProcessOn

Demand

update catalogue with reference to AST_09T

359625-EMD-002, Rev. 02

ASTER: PGE Execution and Output Insertion Process

ACT PGE is successfully executed and newly created AST_09T(L2 Surface Radiance TIR) granule is archived; inventory is updated.


SDSRVDDIST

STMGT

Inven-toryDB

Collection


Client


SBSRV


abc

def

ghi

jkl

mno


0110001011001011001011010111010110001011

1. Insert(AST_09T)

2. Store (AST_09T files)

3. Add (AST_09T metadata)

Data MgmtSubsystem

DDICT


(OMS)


360625-EMD-002, Rev. 02


0110001011001011001011010111010110001011






EcDpAtOpDbGui



Check SoftwareTools



Insert/AcquireTools






for SSI&T)


ExecutionManagement


PGEManagement


DeletionServer

EcDpPrDeletion



AutoSysCOTS Product

QAMonitor

EcDpPrQaMonitorGUI

DataStore









SDSRVDDIST

STMGT

Inven-toryDB

Collection




MediaServer Process


EcDsStCopyServer

Pull MonitorProcess



EcDsStmgtGui




ScienceData Server






EcDsGranuleDelete



EcDsSCLI


DataDistribution



EcDsDdistGui



EcDsDdDCLI

ASTER: CSCI/Component Role in PGE Execution and Output Insertion

Database; receives and stores metadata

Requests STMGT to insert output granule files; adds metadata to inventory

3

4

Reads files from DPS

disks and stores

files in Archive

5

Monitors execution of

PGE; places output files

in output directory

1

2

Handles requests to the Archive for inserting granules


361625-EMD-002, Rev. 02


PRODUCTION

Subscribe


Subscription

Store

ArchiveData

LocateData

CatalogueData

RetrieveData

ProcessOn

Demand

Insert terminates with an insert event notification to Subscribe,which in turn triggers initiation of ETS algorithm

362625-EMD-002, Rev. 02

ASTER: Notification and Subscription Triggering Process

Notify all AST_09T:Insert subscribers. This includes notification of thePlanning Subsystem, for chained processing.


Client

1. TriggerAST_09T:Insertevent

2. Notify (AST_09T:Insert,direct)


0110001011001011001011010111010110001011


SDSRVDDIST

STMGT

Inven-toryDB

Collection

3. Data ProcessingRequest (ETS)


SBSRV


abc

def

ghi

jkl

mno

Data MgmtSubsystem

DDICT


(OMS)


363625-EMD-002, Rev. 02


DomainName Server

resolver



ASTER DARGateway

EcGwDARServer


EcCsEmailParser

FTPNotification

CsFtFTPNotify

Bulk DataServer

BDSpro

NetworkFile System

NFS

CCSMiddleware


SubscriptionServer



FTP


VirtualTerminal

VT


InterfaceCMI





MOJOGateway

EcCsMojoGateway

ConfigurationRegistry Srvr.

EcCsRegistryServer


EcCsMtMGateway


0110001011001011001011010111010110001011






EcDpAtOpDbGui



Check SoftwareTools



Insert/AcquireTools






for SSI&T)


ExecutionManagement


PGEManagement


DeletionServer

EcDpPrDeletion



AutoSysCOTS Product

QAMonitor

EcDpPrQaMonitorGUI

DataStore









SDSRVDDIST

STMGT

Inven-toryDB

Collection




MediaServer Process


EcDsStCopyServer

Pull MonitorProcess



EcDsStmgtGui




ScienceData Server






EcDsGranuleDelete



EcDsSCLI


DataDistribution



EcDsDdistGui



EcDsDdDCLI


DataStore





EcPlPrEditor_IF


EcPlWbEcPlTl



EcPlOdMgr



EcPlProdStrat


EcPlPRGenerator

PlanningSubsystem

(PLS) Planning

abcdefghijklmno

ASTER: CSCI/Component Role in Notification/Subscription Triggering

Database; contains

metadata for search

and retrieval2

Triggers event; retrieves metadata for

requested input granules from the

Sybase/SQS database1

Receives eventnotice from SDSRV;queries database;notifies Planning

3

Begins ETS staging andprocessing activities 6

Adds UR for new granule (AST_09T)to data context; matches metadata against job input needs; notifies Job Management to release ETS

4

Releas

es ETS

job

5

364625-EMD-002, Rev. 02


PRODUCTION

Subscribe


Subscription

Store

ArchiveData

LocateData

CatalogueData

RetrieveData

Retrieve AST_ANC (ASTER ancillary data set) granule as input to ETS production job; PGE execution begins

ProcessOn

Demand

365625-EMD-002, Rev. 02

ASTER: Job (ETS) Staging Process

ETS production job retrieves required AST_ANC(ASTER ancillary data set) input data granule(Note: AST_09T is already available on DPS resources).


SDSRVDDIST

STMGT

Inven-toryDB

Collection


Client


SBSRV


abc

def

ghi

jkl

mno

1. Acquire (AST_ANC,ftpPush)


0110001011001011001011010111010110001011

2. Distribution Request (AST_ANC file, ftpPush)

3. Copy (AST_ANCfile, ftpPush)

4. ftpPush (AST_ANCfile)

Data MgmtSubsystem

DDICT


(OMS)


366625-EMD-002, Rev. 02


0110001011001011001011010111010110001011






EcDpAtOpDbGui



Check SoftwareTools



Insert/AcquireTools






for SSI&T)


ExecutionManagement


PGEManagement


DeletionServer

EcDpPrDeletion



AutoSysCOTS Product

QAMonitor

EcDpPrQaMonitorGUI

DataStore









SDSRVDDIST

STMGT

Inven-toryDB

Collection




MediaServer Process


EcDsStCopyServer

Pull MonitorProcess



EcDsStmgtGui




ScienceData Server






EcDsGranuleDelete



EcDsSCLI


DataDistribution



EcDsDdistGui



EcDsDdDCLI

ASTER: CSCI/Component Rolein Job (ETS) Staging

Allocates space in response to Distribution Requests 6

Releases and

sequencesDPRs (Jobs)

1

Requests staging disks for granule files; requests granule file retrieval by STMGT

5

Stages granule files to read-only cache disks

7


Provides Resource

Manager for ftpPushes

8

Controls staging and processing activities; requests MCF; handles requests to the Archive for retrieving granules

2


Provides Metadata Configuration File (MCF); retrieves metadata; requests metadata Staging Disks; creates and passes distribution requests and metadata files to DDIST

4

3

367625-EMD-002, Rev. 02


PRODUCTION

Subscribe


Subscription

Store

ArchiveData

LocateData

CatalogueData

RetrieveData

Archive newly created AST_08 (L2 Surface Temperature) and AST_05 (L2 Surface Emissivity) granules after completion of ETS PGE;

ProcessOn

Demand

update catalogue with references to AST_08 and AST_05

368625-EMD-002, Rev. 02

ASTER: PGE (ETS) Execution and Output Insertion Process

ETS PGE is successfully executed and newly created AST_08(L2 Surface Temperature) and AST_05 (L2 Surface Emissivity)granules are archived; inventory is updated.


SDSRVDDIST

STMGT

Inven-toryDB

Collection


Client


SBSRV


abc

def

ghi

jkl

mno


0110001011001011001011010111010110001011

1. Insert(AST_08& AST_05)

2. Store (AST_08 &AST_05 files)

3. Add (AST_08 & AST_05 metadata)

Data MgmtSubsystem

DDICT


(OMS)


369625-EMD-002, Rev. 02


0110001011001011001011010111010110001011






EcDpAtOpDbGui



Check SoftwareTools



Insert/AcquireTools






for SSI&T)


ExecutionManagement


PGEManagement


DeletionServer

EcDpPrDeletion



AutoSysCOTS Product

QAMonitor

EcDpPrQaMonitorGUI

DataStore









SDSRVDDIST

STMGT

Inven-toryDB

Collection




MediaServer Process


EcDsStCopyServer

Pull MonitorProcess



EcDsStmgtGui




ScienceData Server






EcDsGranuleDelete



EcDsSCLI


DataDistribution



EcDsDdistGui



EcDsDdDCLI

ASTER: CSCI/Component Role in PGE Execution and Output Insertion

Reads files from DPS

disks and stores

files in Archive

5

Monitors execution of

PGE; places output

files in output directory

1

Database; receives and stores metadata

Requests STMGT to insert output granule files; adds metadata to inventory

3

4

2

Handles requests to the Archive for inserting granules


370625-EMD-002, Rev. 02


Insert terminates with an insert event notification to Subscribe,

RETRIEVAL

Deliver

Electronic

Search& Order

OrderProducts

Subscribe

Notification StandingOrders

Store

ArchiveData

LocateData

CatalogueData

RetrieveData

which triggerse-mail notification to the Science User that the AST_08 granule has been inserted; standing order processing for new AST_08 granule can begin

371625-EMD-002, Rev. 02

ASTER: Notification & Subscription Triggering (AST_08) Process

With insertion of the AST_08 (L2 Surface Temperature) granule,the ASTER Scientist is notified by e-mail; processing can beginfor the standing order


Client


0110001011001011001011010111010110001011

1. TriggerAST_08:Insertevent


SDSRVDDIST

STMGT

Inven-toryDB

Collection


SBSRV


abc

def

ghi

jkl

mno

2. Notify (AST_08:Insert,e-mail)

Data MgmtSubsystem

DDICT


(OMS)


372625-EMD-002, Rev. 02


DomainName Server

resolver



ASTER DARGateway

EcGwDARServer


EcCsEmailParser

FTPNotification

CsFtFTPNotify

Bulk DataServer

BDSpro

NetworkFile System

NFS

CCSMiddleware


SubscriptionServer



FTP


VirtualTerminal

VT


InterfaceCMI





MOJOGateway

EcCsMojoGateway


EcCsRegistryServer


EcCsMtMGateway


SDSRVDDIST

STMGT

Inven-toryDB

Collection




MediaServer Process


EcDsStCopyServer

Pull MonitorProcess



EcDsStmgtGui




ScienceData Server






EcDsGranuleDelete



EcDsSCLI


DataDistribution



EcDsDdistGui



EcDsDdDCLI

ASTER: CSCI/Component Role in Notification/Subscription Trigger

Sendmail deliverse-mail messageto mail host

3

Receives event notice from SDSRV; queries database; executes e-mail notification

2

Triggers event (AST_08:Insert)1

373625-EMD-002, Rev. 02


Submit acquire request for new AST_08granule on behalf of scientist

RETRIEVAL

Deliver

Electronic

Search& Order

OrderProducts

Store

ArchiveData

LocateData

CatalogueData

RetrieveData

Subscribe


374625-EMD-002, Rev. 02

ASTER: Standing Order, Acquire Submission Process

Subscription Server submits acquire request for AST_08 (L2 SurfaceTemperature) data, via ftpPush, on behalf of the Science User


Client


0110001011001011001011010111010110001011


SDSRVDDIST

STMGT

Inven-toryDB

Collection


SBSRV


abc

def

ghi

jkl

mno

1. Acquire (AST_08:ftpPush)

Data MgmtSubsystem

DDICT


(OMS)


375625-EMD-002, Rev. 02


DomainName Server

resolver



ASTER DARGateway

EcGwDARServer


EcCsEmailParser

FTPNotification

CsFtFTPNotify

Bulk DataServer

BDSpro

NetworkFile System

NFS

CCSMiddleware


SubscriptionServer



FTP


VirtualTerminal

VT


InterfaceCMI





MOJOGateway

EcCsMojoGateway


EcCsRegistryServer


EcCsMtMGateway


SDSRVDDIST

STMGT

Inven-toryDB

Collection




MediaServer Process


EcDsStCopyServer

Pull MonitorProcess



EcDsStmgtGui




ScienceData Server






EcDsGranuleDelete



EcDsSCLI


DataDistribution



EcDsDdistGui



EcDsDdDCLI

ASTER: CSCI/Component Role in Standing Order, Acq. Submission

Retrieves metadata

Database; stores metadata for search and retrieval

3

2

Connects to SDSRV; establishes data context; submits Acquire request for ftpPush of granule and request for e-mail notice of distribution

1

376625-EMD-002, Rev. 02


Retrieve newly created AST_08 granule

RETRIEVAL

Deliver

Electronic

Store

ArchiveData

LocateData

CatalogueData

RetrieveData

Subscribe


Search& Order

OrderProducts

377625-EMD-002, Rev. 02

ASTER: Retrieval of Data for Distribution Process

Retrieve newly created AST_08 (L2 Surface Temperature) granulefrom the Archive and update the order tracking information.


Client


0110001011001011001011010111010110001011


SDSRVDDIST

STMGT

Inven-toryDB

Collection


SBSRV


abc

def

ghi

jkl

mno

1. Distribution Request (AST_08:ftpPush)

2. Copy (AST_08, ftpPush)


MSS3. Update Order

(AST_08) Data Mgmt

SubsystemDDICT


(OMS)


378625-EMD-002, Rev. 02


SDSRVDDIST

STMGT

Inven-toryDB

Collection




MediaServer Process


EcDsStCopyServer

Pull MonitorProcess



EcDsStmgtGui




ScienceData Server






EcDsGranuleDelete



EcDsSCLI


DataDistribution



EcDsDdistGui



EcDsDdDCLI


(MSS)





















(MLCI)


Remedy ARS (COTS)


SoftwareLicense Mgr


ASTER: CSCI/Component Role in Retrieval of Data for Distribution


2

Requests meta-

data Staging

Disks;

creates and

passes

distributio

n

request

and metadata

files to

DDIST

1

Allocates space

in response to

Distribution Requests 4 Requests

staging disks for granule files; requests granule file retrieval by STMGT; updates order

3

Stages granulefiles to read-only cache disks

5

Provides Resource

Manager for

ftpPushes

6

Updates order

tracking data

7


379625-EMD-002, Rev. 02


Ftp newly created AST_08 granule to scientist’s workstation and send ane-mail notification of the distribution

RETRIEVAL

Subscribe


Search& Order

OrderProducts

Deliver

Electronic

Store

ArchiveData

LocateData

CatalogueData

RetrieveData

380625-EMD-002, Rev. 02

ASTER: Electronic Data Push Distribution Process

Ftp AST_08 (L2 Surface Temperature) granule toASTER Scientist’s workstation.


Client


0110001011001011001011010111010110001011


SDSRVDDIST

STMGT

Inven-toryDB

Collection


abc

def

ghi

jkl

mno

1. Ftp AST_08 data

2. Build e-mailnotification

3. Send e-mailnotification ofAST_08 distribution


FTP / E-Mail

Data MgmtSubsystem

DDICT


(OMS)


381625-EMD-002, Rev. 02


DomainName Server

resolver



ASTER DARGateway

EcGwDARServer


EcCsEmailParser

FTPNotification

CsFtFTPNotify

Bulk DataServer

BDSpro

NetworkFile System

NFS

CCSMiddleware


SubscriptionServer



FTP


VirtualTerminal

VT


InterfaceCMI





MOJOGateway

EcCsMojoGateway


EcCsRegistryServer


EcCsMtMGateway


SDSRVDDIST

STMGT

Inven-toryDB

Collection




MediaServer Process


EcDsStCopyServer

Pull MonitorProcess



EcDsStmgtGui




ScienceData Server






EcDsGranuleDelete



EcDsSCLI


DataDistribution



EcDsDdistGui



EcDsDdDCLI

ASTER: CSCI/Component Role in Electronic Data Push Distribution

Client for CSS

execution of actual

ftp

1

Builds e-mail notifica-tion that user’s order has been filled;sends it to the scientist

3

Executes ftp 2


4

382625-EMD-002, Rev. 02


RETRIEVAL

Subscribe


Search& Order

OrderProducts

Deliver

Electronic

Store

ArchiveData

LocateData

CatalogueData

RetrieveData

Scientist uses EOSView to perform Quality Assurance (QA) check on the AST_08 product and e-mails DAAC Operations; DAAC Operations updates AST_08 inventory metadata

383625-EMD-002, Rev. 02

ASTER: QA Metadata Update Process

Science User uses EOSView tool to review AST_08 (L2 SurfaceTemperature) product and sends e-mail request to DAAC Operationsfor update of the Science QA Metadata for the granule; DAAC Operationsuses QA Monitor tool for the update.


Client


SDSRVDDIST

STMGT

Inven-toryDB

Collection


SBSRV


abc

def

ghi

jkl

mno

1. Invoke EOSViewto view AST_08


0110001011001011001011010111010110001011

2. Scientist e-mails DAAC withrequest to update the Science QAmetadata for the granule

3. InvokeQA Monitor

4. Request update ofQA metadata

5. Update metadata inventory, AST_08

Data MgmtSubsystem

DDICT


(OMS)


384625-EMD-002, Rev. 02


0110001011001011001011010111010110001011






EcDpAtOpDbGui



Check SoftwareTools



Insert/AcquireTools






for SSI&T)


ExecutionManagement


PGEManagement


DeletionServer

EcDpPrDeletion



AutoSysCOTS Product

QAMonitor

EcDpPrQaMonitorGUI

DataStore









SDSRVDDIST

STMGT

Inven-toryDB

Collection




MediaServer Process


EcDsStCopyServer

Pull MonitorProcess



EcDsStmgtGui




ScienceData Server






EcDsGranuleDelete



EcDsSCLI


DataDistribution



EcDsDdistGui



EcDsDdDCLI

ASTER: CSCI/Component Role inQA Metadata Update

Submits request to

SDSRV for update

of granule metadata

2



(ODFRM) CSCI

HTML UserInterface

ODFRM

CGIPrograms

EcClOdRequest

Desktop(DESKT) CSCI

User ProfileGateway




EOSView

ASTER DARTool

EcClWbJdt

Generates query calls to inventory database; packages and returns results; updates metadata

Database; contains metadata 4

3

Displays HDF files

for visualization 1

385625-EMD-002, Rev. 02

ASTER Scenario: Expedited DataIN

SERT

ION

RETRIEVAL

PRODUCTION


Store


Deliver

Generate Plan

Ingest

Data SubscriptionData InsertionData Notification

386625-EMD-002, Rev. 02

INSE

RTIO

N

Ingest

Polling

DAAC submits subscription, on behalf of scientist for notification of new ASTER Expedited data

Expedited Data

Subscribe


Subscription

Store

ArchiveData

LocateData

CatalogueData

RetrieveData

387625-EMD-002, Rev. 02

ASTER: User Subscription Registration for AST_EXP Process

DAAC enters subscription, on behalf of scientist, for e-mail notificationof newly inserted ASTER Expedited Data product.


SDSRVDDIST

STMGT

Inven-toryDB

Collection



abc

def

ghi

jkl

mno


0110001011001011001011010111010110001011

Client

1. Subscribe (AST_EXP, e-mail)Communications

SubsystemRegistry

2. Get distribution options


(SSS)

Data MgmtSubsystem

DDICT


(OMS)


388625-EMD-002, Rev. 02


DomainName Server

resolver



ASTER DARGateway

EcGwDARServer


EcCsEmailParser

FTPNotification

CsFtFTPNotify

Bulk DataServer

BDSpro

NetworkFile System

NFS

CCSMiddleware


SubscriptionServer



FTP


VirtualTerminal

VT


InterfaceCMI





MOJOGateway

EcCsMojoGateway


EcCsRegistryServer


EcCsMtMGateway



NBSRV GUIEcNbGUI




SSS (NBSRV)Database




(SSS)

ASTER: CSCI/Component Role in AST_EXP Subscription Registration

GUI used by User ServicesRepresentative to createsubscription for Science User

1

Stores subscription 3

Providesdistributionoptions

2

389625-EMD-002, Rev. 02

INSE

RTIO

N

Ingest

Polling

EDOS initiates AST_EXP (Expedited Data) ingestion via EDOSExpedited Data Set Delivery Record (EDR)

Expedited Data (Cont.)

Subscribe


Subscription

Store

ArchiveData

LocateData

CatalogueData

RetrieveData

EDOS

390625-EMD-002, Rev. 02

ASTER: Polling Ingest for AST_EXP Process

Ingest begins polling specified location (directory), looking for an Expedited Data Set Delivery Record (EDR) file.


SDSRVDDIST

STMGT

Inven-toryDB

Collection



abc

def

ghi

jkl

mno


0110001011001011001011010111010110001011

Client


SBSRV

1. Ingest Subsystempolling for files

EDOS2. Move EDR

file todirectory

3. Read EDR file

4. Get mcf (AST_EXP)

5. Read AST_EXPfiles

EDOS transfers the file, via ftp, to thepredetermined directory. (Location, directory, username, and passwordare as specified in the EDOS-ECS Operations Agreement.)

Data MgmtSubsystem

DDICT


(OMS)


391625-EMD-002, Rev. 02

IngestSubsystem

(INS) Ingest

Ingest(INGST) CSCI


ProcessEcInPolling


ProcessEcInGUI

EcInTapeReaderGUI


ProcessEcInReqMgr


ProcessEcInGran

IngestDatabase



EcInEmailGWServer


SDSRVDDIST

STMGT

Inven-toryDB

Collection




MediaServer Process


EcDsStCopyServer

Pull MonitorProcess



EcDsStmgtGui




ScienceData Server






EcDsGranuleDelete



EcDsSCLI


DataDistribution



EcDsDdistGui



EcDsDdDCLI

ASTER: CSCI/Component Role in Polling Ingest for AST_EXP

Polls remote location for files named *.EDR.XFR; packages requests

1

PackagesIngest Requestinto Granules

4

Staging Disk Server

receives request from

Ingest to create staging

disk and reserves space

5 Provides metadata configuration file (mcf)

3

Ingest reads delivery

record file and data files

from provider site via ftp 6

Requests metadata configuration file (mcf) from SDSRV; requests staging disk2


392625-EMD-002, Rev. 02

INSE

RTIO

N

Ingest

Polling


Subscribe


Subscription

Store

ArchiveData

LocateData

CatalogueData

RetrieveData

Archive ingested AST_EXP data; update catalogue with reference to AST_EXP

393625-EMD-002, Rev. 02

ASTER: AST_EXP Insertion Process


SDSRVDDIST

STMGT

Inven-toryDB

Collection



abc

def

ghi

jkl

mno


0110001011001011001011010111010110001011

Client


SBSRV

Newly ingested AST_EXP granule is archived; inventory is updated.

1. Insert (AST_EXP)

2. Store (AST_EXP files)

3. Add (AST_EXP metadata)

Data MgmtSubsystem

DDICT


(OMS)


394625-EMD-002, Rev. 02

IngestSubsystem

(INS) Ingest

Ingest(INGST) CSCI


ProcessEcInPolling


ProcessEcInGUI

EcInTapeReaderGUI


ProcessEcInReqMgr


ProcessEcInGran

IngestDatabase



EcInEmailGWServer


SDSRVDDIST

STMGT

Inven-toryDB

Collection




MediaServer Process


EcDsStCopyServer

Pull MonitorProcess



EcDsStmgtGui




ScienceData Server






EcDsGranuleDelete



EcDsSCLI


DataDistribution



EcDsDdistGui



EcDsDdDCLI

ASTER: CSCI/Component Rolein AST_EXP Insertion

Stores va

lidate

d

metadata

in th

e

inventory

4

Validates metadata;

requests archiving of

granule files; adds

metadata to inventory

2

Builds metadata

file; requests

SDSRV to

validate

metadata;

requests

SDSRV to insert

received files

1


Reads inserted

files from Ingest

staging disks

3

395625-EMD-002, Rev. 02

INSE

RTIO

N

Ingest

Polling


Subscribe


Subscription

Store

ArchiveData

LocateData

CatalogueData

RetrieveData

Insert in the SDSRV database triggers an event in the NBSRV database. Subscribe e-mails ASTER Scientist notice of a completed AST_EXP granule insert.

396625-EMD-002, Rev. 02

ASTER: AST_EXP Event Notification Process

Notify all AST_EXP:Insert event subscribers.


E-Mail


abc

def

ghi

jkl

mno


0110001011001011001011010111010110001011

Client


1. Trigger(AST_EXP:Insert)


SDSRVDDIST

STMGT

Inven-toryDB

Collection

2. Notify (AST_EXP:Insert,e-mail)


(SSS)Data Mgmt

SubsystemDDICT


(OMS)


397625-EMD-002, Rev. 02


DomainName Server

resolver



ASTER DARGateway

EcGwDARServer


EcCsEmailParser

FTPNotification

CsFtFTPNotify

Bulk DataServer

BDSpro

NetworkFile System

NFS

CCSMiddleware


SubscriptionServer



FTP


VirtualTerminal

VT


InterfaceCMI





MOJOGateway

EcCsMojoGateway


EcCsRegistryServer


EcCsMtMGateway


SDSRVDDIST

STMGT

Inven-toryDB

Collection




MediaServer Process


EcDsStCopyServer

Pull MonitorProcess



EcDsStmgtGui




ScienceData Server






EcDsGranuleDelete



EcDsSCLI


DataDistribution



EcDsDdistGui



EcDsDdDCLI

ASTER: CSCI/Component Rolein AST_EXP Event Notification


4

AST_EXP:Insert triggers

event; places the event in

the NBSRV database

1



NBSRV GUIEcNbGUI




SSS (NBSRV)Database




(SSS) Detects event; gets metadata from SDSRV

database and stores it in NBSRV database;

matches event to subscription and places

subscription into the Action Queue

2

Detects

subscription;

composes and

sends e-mail

notification of

the AST_EXP:Insert

event3

398625-EMD-002, Rev. 02

User Registration

Science User starts the form for first-time User Registration from the EOS Data Gateway Web Client and submits registration information.

RETRIEVAL

UserAccess

UserRegistrationSubscribe

Deliver

ManageSystemMonitor &Coordinate

ManageApplications

Logistics

399625-EMD-002, Rev. 02

User Registration Request Process

Scientist invokes the form for first-time User Registration from the EOS DataGateway web client and then enters and submits registration data.



abc

def

ghi

jkl

mno


0110001011001011001011010111010110001011

Client



SDSRVDDIST

STMGT

Inven-toryDB

Collection 1. Request UserRegistration


(MSS)Data Mgmt

SubsystemDDICT2. Submit

RegistrationRequest Order Manager

Subsystem(OMS)


400625-EMD-002, Rev. 02


(MSS)


(MLCI)


Remedy ARS (COTS)


SoftwareLicense Mgr






TCP Wrappers (COTS)Tripwire (COTS)

Crack (COTS)














CSCI/Component Role inUser Registration Request



(ODFRM) CSCI

HTML UserInterface

ODFRM

CGIPrograms

EcClOdRequest

Desktop(DESKT) CSCI

User ProfileGateway




EOSView

ASTER DARTool

EcClWbJdt

Receives request from EDG and submits it to User Registration Server (at SMC)

1

Saves request as a new User

Profile at SMC, for replication

to each remote DAAC2

401625-EMD-002, Rev. 02

User Registration (Cont.)

Science User calls home DAAC Operations (User Services) to request any needed privileges (e.g., access to restricted granules, ASTER DAR privileges); DAAC Operations logs into SMC remotely, retrieves User Profile, and assigns approved privileges

RETRIEVAL

UserAccess



ManageApplications

Logistics

Deliver

402625-EMD-002, Rev. 02

User Registration Approval Process

DAAC User Services Representative at home DAAC uses User RegistrationServer Graphical User Interface (GUI) at SMC to retrieve User Profile andupdate the MSS User Profile approved privileges, which are then replicated.



abc

def

ghi

jkl

mno


0110001011001011001011010111010110001011

Client



SDSRVDDIST

STMGT

Inven-toryDB

Collection

1. Retrieve UserProfile andupdate theprivileges


(MSS)Data Mgmt

SubsystemDDICT


(OMS)


403625-EMD-002, Rev. 02


(MSS)


(MLCI)


Remedy ARS (COTS)


SoftwareLicense Mgr







Crack (COTS)














CSCI/Component Role inUser Registration Approval

GUI at SMC retrieves User

Profile; operator updates User

Profile information; server

stores the updated User

Profile, which is replicated for

read-only access at all sites

404625-EMD-002, Rev. 02

User Registration (Cont.)

RETRIEVAL

UserAccess



ManageApplications

Logistics

Deliver

System Management Subsystem (MSS) sends e-mail to the Science User confirming registration and anyspecial privileges

405625-EMD-002, Rev. 02

User Registration Confirmation Process

System Management Subsystem (MSS) User Registration Servercreates and sends e-mail to the Science User confirming registration andany special privileges.


E-Mail


abc

def

ghi

jkl

mno


0110001011001011001011010111010110001011

Client



SDSRVDDIST

STMGT

Inven-toryDB

Collection

1. Build user e-mailmessage

2. E-mail notificationSystems MgmtSubsystem

(MSS)

Data MgmtSubsystem

DDICT


(OMS)


406625-EMD-002, Rev. 02


(MSS)


(MLCI)


Remedy ARS (COTS)


SoftwareLicense Mgr







Crack (COTS)















DomainName Server

resolver



ASTER DARGateway

EcGwDARServer


EcCsEmailParser

FTPNotification

CsFtFTPNotify

Bulk DataServer

BDSpro

NetworkFile System

NFS

CCSMiddleware


SubscriptionServer



FTP


VirtualTerminal

VT


InterfaceCMI





MOJOGateway

EcCsMojoGateway


EcCsRegistryServer


EcCsMtMGateway

CSCI/Component Role inUser Registration Confirmation

User Registration Server builds

e-mail message confirming

registration and privileges1

Sends e-mail notification confirming registrationand privileges

2

407625-EMD-002, Rev. 02

MODIS Scenario

MODIS Goals- Standing Order Submittal- Polling Ingest- Standard Production and Chaining- Data Access

MODIS PreconditionsMODIS ESDTs Inserted into ECS- MOD000, MOD01, MOD02, MOD03, MOD07, MOD35,MODVOLC, MOD35ANC, GDAS_0ZF, Others

MODIS PGEs passed SSI&T and installed- PGE01, PGE02, PGE03

Ancillary and static data inserted into Data Server

MODIS

Automatic PGE Scheduling

Chaining of PGEs

Standing Orders

408625-EMD-002, Rev. 02

MODIS Scenario: Data AccessIN

SERT

ION

RETRIEVAL

PRODUCTION


Store


Deliver

Generate Plan

Ingest

Search & Order Granules

409625-EMD-002, Rev. 02

MODIS: Data Access

RETRIEVAL

Subscribe

Science User searches for MODIS granules that are inECS data holdings

Search& OrderInventorySearch

OrderProducts

Deliver

Electronic

Media

StoreLocateData

RetrieveData

410625-EMD-002, Rev. 02

MODIS: Inventory Search Process

Science User uses EOS Data Gateway (EDG) Web Client to perform an InventorySearch for MOD07_L2 data (MODIS Level 2 Joint Atmosphere Product of profiles, ozone, water, and other components) over a specified spatial/temporaldomain.

1. Search forMOD07_L2granules


SDSRVDDIST

STMGT

Inven-toryDB

Collection



abc

def

ghi

jkl

mno


0110001011001011001011010111010110001011

Client


E-Mail

2. Submit query (MOD07_L2,space&time)

3. Submit translatedquery to SDSRV

EDG

UserAuthentication

Data MgmtSubsystem

DDICT


(OMS)


411625-EMD-002, Rev. 02

Data MgmtSubsystem

(DMS)



EcDmMaintTool




EcDmDictService



EcDmV0ToECSGateway

MODIS: CSCI/Component Rolein Inventory Search

Translates search criteria to a search object and submits query to SDSRV Search service

3



(ODFRM) CSCI

HTML UserInterface

ODFRM

CGIPrograms

EcClOdRequest

Desktop(DESKT) CSCI

User ProfileGateway




EOSView

ASTER DARTool

EcClWbJdt

Provides user authentication and privileges

2


(MSS)





















(MLCI)


Remedy ARS (COTS)


SoftwareLicense Mgr


Holds User Profile information 1

412625-EMD-002, Rev. 02

MODIS: Data Access (Cont.)

Granules of interest are locatedin the ECS data holdings

RETRIEVAL

Subscribe


OrderProducts

Deliver

Electronic

Media

StoreLocateData

RetrieveData

413625-EMD-002, Rev. 02

MODIS: Data Search Process

SDSRV queries Sybase/SQS database for granules meeting search criteriaand returns results.


SDSRVDDIST

STMGT

Inven-toryDB

Collection



abc

def

ghi

jkl

mno


0110001011001011001011010111010110001011

1. Query Sybase

Client

CommunicationsSubsystemRegistry

2. Package andreturn results 4. Return translated results for

display to the user

3. Retrieve mediaoptions fromRegistry Server

Data MgmtSubsystem

DDICT


(OMS)


414625-EMD-002, Rev. 02


DomainName Server

resolver



ASTER DARGateway

EcGwDARServer


EcCsEmailParser

FTPNotification

CsFtFTPNotify

Bulk DataServer

BDSpro

NetworkFile System

NFS

CCSMiddleware


SubscriptionServer



FTP


VirtualTerminal

VT


InterfaceCMI





MOJOGateway

EcCsMojoGateway


EcCsRegistryServer


EcCsMtMGateway


SDSRVDDIST

STMGT

Inven-toryDB

Collection




MediaServer Process


EcDsStCopyServer

Pull MonitorProcess



EcDsStmgtGui




ScienceData Server






EcDsGranuleDelete



EcDsSCLI


DataDistribution



EcDsDdistGui



EcDsDdDCLI

MODIS: CSCI/Component Rolein Data Search

Data MgmtSubsystem

(DMS)



EcDmMaintTool




EcDmDictService



EcDmV0ToECSGatewayBreaks down Query and generates calls to inventory database; packages and returns results


1

2

Provides

media options

associated

with archived

data to V0

Gateway3

Translates results and returns them to the EOS Data Gateway Web Client for display to the user

4

415625-EMD-002, Rev. 02


Science User decides to ordergranules of interest

RETRIEVAL

Subscribe


OrderProducts

Deliver

Electronic

Media

StoreLocateData

RetrieveData

416625-EMD-002, Rev. 02

CommunicationsSubsystemRegistry



0110001011001011001011010111010110001011

Client


abc

def

ghi

jkl

mno

2. Submit order(MOD07_L2, ftpPush)

EDG

UserAuthentication

1. Order MOD07_L2granules

Science User uses EOS Data Gateway Web Client to order MOD07_L2 Granules via ftp push.


SDSRVDDIST

STMGT

Inven-toryDB

Collection

3. InsertProductDistributionRequest

Data MgmtSubsystem

DDICT


Data Pool(DPL)

DATAPOOL


(OMS)

MODIS: Order MOD07_L2

417625-EMD-002, Rev. 02


(MSS)





















(MLCI)


Remedy ARS (COTS)


SoftwareLicense Mgr


MODIS: CSCI/Component Role in Order MOD07_L2

Data MgmtSubsystem

(DMS)



EcDmMaintTool




EcDmDictServer


V0 to ECSGateway ServerEcDmV0ToECSGateway

Passes request forMOD07_L2 granules to Order Manager Database

4



(ODFRM) CSCI

HTML UserInterface

ODFRM

CGIPrograms

EcClOdRequest

Desktop(DESKT) CSCI

User ProfileGateway



User RegistrationTool (URT) Scripts

EcClWbUrEcClWbUrConfirmEcClWbUrSubmit


EOSView

ASTER DARTool

EcClWbJdt




Order ManagerServer

EcOmOrderManager




(OMS)


ProductionModule


Provides ECS names for V0 terms

3

Stores request in database

5

Transmits User Profile

to EOS Data Gateway

Web Client

1

Provides UserProfile2

418625-EMD-002, Rev. 02

Ftp data to Science User;notify Science User ofdistribution


RETRIEVAL

Subscribe


OrderProducts

Deliver

Electronic

Media

StoreLocateData

RetrieveData

419625-EMD-002, Rev. 02



0110001011001011001011010111010110001011

Client


abc

def

ghi

jkl

mno

EDG

Push MOD07_L2 data to Science User’s workstation. Send e-mail to Science User with notification of distribution.


SDSRVDDIST

STMGT

Inven-toryDB

Collection


5. Ftp MOD07_L2data


E-Mail

1. Submit granule staging requests2. Requestdata insert toData Pool


Data Pool(DPL)

DATAPOOL

4. Copy granules from Data Pool

7. Send e-mail notificationof MOD07_L2 distribution


(OMS)

3. Copy granules

MODIS: Distribute MOD07_L2

420625-EMD-002, Rev. 02


DomainName Server

resolver



ASTER DARGateway

EcGwDARServer


EcCsEmailParser

FTPNotification

CsFtFTPNotify

Bulk DataServer

BDSpro

NetworkFile System

NFS

CCSMiddleware


SubscriptionServer



FTP


VirtualTerminal

VT


InterfaceCMI





MOJOGateway

EcCsMojoGateway


EcCsRegistryServer


EcCsMtMGateway

MODIS: CSCI/Component Role in Distribute MOD07_L2


SDSRVDDIST

STMGT

Inven-toryDB

Collection




MediaServer Process


EcDsStCopyServer

Pull MonitorProcess



EcDsStmgtGui




ScienceData Server






EcDsGranuleDelete



EcDsSCLI


DataDistribution



EcDsDdistGui



EcDsDdDCLI

Data Pool(DPL)

DATAPOOL


COTS FTPServer

wuftp


EcDlWebAccess

Data PoolDatabase


Data PoolInsert UtilityEcDlInsertUtilityEcDlActionDriverEcDlQuickServer

EcDlM2XTApp












Data PoolCollection









Order ManagerServer

EcOmOrderManager




(OMS)


ProductionModule


Insert granule staging requests

2

Fetch request forMOD07_L2 granules from Order Manager Database; submit granule staging requests; request ftp push; request e-mail support

1

Request data insert to Data Pool from cache or archive

3

Copy granules to Data Pool4


5

421625-EMD-002, Rev. 02


RETRIEVAL

Subscribe

Science User searches for MODIS granules that are inthe Data Pool


OrderProducts

Deliver

Electronic

Media

StoreLocateData

RetrieveData

422625-EMD-002, Rev. 02

MODIS: Data Pool Search and Order ProcessScience User uses the Data Pool Web Access GUI to search for MOD11_L2 data (MODIS/Terra Land Surface Temperature/Emissivity 5-Min L2 Swath 1km)over a specified spatial/temporal domain and submit an order for delivery on DVD.


SDSRVDDIST

STMGT

Inven-toryDB

Collection



abc

def

ghi

jkl

mno


0110001011001011001011010111010110001011

Client


E-Mail

1. Submit query (MOD11_L2,space&time)


4. Select mediaoption

2. Request mediaoptions

5. Submit hard media order

(MOD11_L2, DVD)

3. Report media options


(OMS)

Data Pool(DPL)

DATAPOOL


(MSS)

423625-EMD-002, Rev. 02

Data Pool(DPL)

DATAPOOL


COTS FTPServer

wuftp


EcDlWebAccess

Data PoolDatabase


Data PoolInsert UtilityEcDlInsertUtilityEcDlActionDriverEcDlQuickServer

EcDlM2XTApp












Data PoolCollection






MODIS: CSCI/Component Rolein Data Pool Search and Order




Order ManagerServer

EcOmOrderManager




(OMS)


ProductionModule


Determine media options for specified granule(s);

2

Locate granules of interest; request media options; select media option; submit order

1

Insert order 3

424625-EMD-002, Rev. 02

RETRIEVAL

Subscribe


OrderProducts

StoreLocateData

RetrieveData

Request for physical media distribution of granules is dispatched to the relevant Production Module; granules are copied to DVDand prepared for shipment to scientist


Deliver

Electronic

Media

425625-EMD-002, Rev. 02


SDSRVDDIST

STMGT

Inven-toryDB

Collection



0110001011001011001011010111010110001011

Client


abc

def

ghi

jkl

mno


(OMS)


(MSS)

Data Pool(DPL)

DATAPOOL

Dispatch the validated request for MODIS granules to the appropriate production module. Copy granules from Data Pool to DVD. Prepare shippingdocuments and e-mail notification of shipment. Ship tape to Science User.Send e-mail notification of shipment.

MODIS: DVD Creation/Distribution Process


4. Send e-mail notification

5. Update OrderTracking Status

1. Copy granules



E-Mail

2. Ship DVD

426625-EMD-002, Rev. 02


DomainName Server

resolver



ASTER DARGateway

EcGwDARServer


EcCsEmailParser

FTPNotification

CsFtFTPNotify

Bulk DataServer

BDSpro

NetworkFile System

NFS

CCSMiddleware


SubscriptionServer



FTP


VirtualTerminal

VT


InterfaceCMI





MOJOGateway

EcCsMojoGateway


EcCsRegistryServer


EcCsMtMGateway


(MSS)





















(MLCI)


Remedy ARS (COTS)


SoftwareLicense Mgr


MODIS: CSCI/Component Role in Distribute on Physical Media




Order ManagerServer

EcOmOrderManager




(OMS)


ProductionModule


Posts OrderTracking

status updates from OMS to database

5

Sendmaildeliverse-mail messageto mail host

63

Displays request/volume status; activates PMD requests; marks changes in request status

3

Copies data from Data Pool; creates image files; prints jewel case insert; prints summary

4

Dispatches request to production module; provides order-tracking status updates to MSS; prints packing list; prints shipping labels; prepares Distribution Notice for e-mail

1

Stores PMD request information

2

427625-EMD-002, Rev. 02

Summary

• Thirteen subsystems• Special Access: ASTER

– DAR– Expedited Data Support

• Data Transfer from External Provider to ECS– DTF-2 Tape Ingest– Polling Ingest

• Planning and Data Processing– Standing Orders– On-Demand Processing: Chaining– QA Update

428625-EMD-002, Rev. 02

Summary (Cont.)

• Data Access and Distribution– User Registration– Archive Data Search– Electronic Distribution– Data Pool Search– Media Distribution

429625-EMD-002, Rev. 02

References

• 305-EMD-001, Release 7.10 Segment/Design Specification for the EMD Project*

• 313-EMD-001, Release 7.10 Internal Interface Control Document for the EMD Project*

*Note: These documents are available on EDHS (http://edhs1.gsfc.nasa.gov).

Introduction and Detailed System Overview: Science Data … · Introduction and Detailed System Overview: Science Data Processing Internal Training Introduction and Detailed System

Documents