MIIC-2 Kick-Off Meeting May 28, 2014 Pamela Rinsland Mike Little ATMOSPHERIC SCIENCE DATA CENTER.

MIIC-2 Kick-Off MeetingMay 28, 2014

Pamela RinslandMike Little

ATMOSPHERICSCIENCEDATA CENTER

04/19/232

ATMOSPHERICSCIENCEDATA CENTER

AGENDA

• ASDC Web Services• MIIC-2 Services Architecture• Support Plans and Issues

ASDC Areas of ModernizationFocus on Discovery and Access

Data Stewardship Data Distribution Data Production

ASDC

Cloud ComputingGIS OPeNDAPiRODS

• Partners: AWS, OCIO• Planning: 2013 - present• Experiments: June ‘14• IT Lab “cloudbursting”

(ARC, GSFC)

• Partners: NASA NCCS, RENSI, HIFLD, OPeNDAP• OPenDAP/Giovanni operational; MIIC-1 prototype• GIS experiments close to Operational Status• Parameter selection based on Ontology - ODISSEES• iRODS in progress

Ontology

SemanticTechnology

Service Oriented

Architecture

DigitalLibrary

Items in green are areas in work; Items in yellow are in still in preliminary conceptual planning phase

AnalyticTool

Support

Hadoop

04/19/234

ATMOSPHERICSCIENCEDATA CENTER ArcGIS Prototype with CERES EBAF Surface Data

GIS Use Case CERES Surface Flux with Freshwater Lakes

04/19/236

ATMOSPHERICSCIENCEDATA CENTER MIIC-2 Services Architecture (Functional View)

04/19/237

ATMOSPHERICSCIENCEDATA CENTER MIIC-2 Services Architecture (Hardware View)

04/19/238

• ASDC– Characterize system at ASDC “as is” to determine

bottlenecks and saturation (what if we have 10 calibration plans executing at once?) so we can deploy a responsive MIIC-2 configuration

– Upgrade to NFS over 10GigE– Re-characterize the upgraded system using

AppDynamics to monitor resource consumption– Refine architecture to reflect path to scalability– Identify/acquire MIIC2 services server and MIIC2

services PostgreSQL DB server– Acquire a Load balancer to handle CPU loads on MIIC2

services server(s); i/o loads on OPeNDAP servers

ATMOSPHERICSCIENCEDATA CENTER Support Plans and Issues

04/19/239

• NCDC– Determine available/need for a MIIC-2 deployment

“box-wise” ($15K can get a decent server)– Determine latency drivers: off-campus pipe width

(10Gbps? OC-48? etc.) Internal networking– Confirm target files at NCDC to publish via MIIC-2;

Confirm files at NCDC in a format that can be handled through standard OPeNDAP? (HDF5?)

– Determine what NCDC staff expect to be able to install and verify the MIIC-2 services (Delivery package)

ATMOSPHERICSCIENCEDATA CENTER Support Plans and Issues

04/19/2310

• NASA-LaRC and NCDC - Space Act Agreement

• Determine approach: Umbrella with annex or “one and done” (current draft example)

• ESDIS/ECHO a separate effort– Conversation with Jeff Walter last week (Andy Mitchell was

unavailable) but need to close the loop

• Signature levels required• Use NASA Space Act Agreement Maker (SAAM) tool• Rudimentary milestones required

– MIIC-2 schedule is one candidate, but don’t want to add more bureaucracy

ATMOSPHERICSCIENCEDATA CENTER Support Plans and Issues

04/19/2311

• NASA-LaRC and NCDC – Schedule of Tasks and Performers

• Determine schedule for deliverables and activities to support ACCESS proposal 2-year period of performance

• Technical POCs for tasks• Communication approach within team (internal and

ahead of required ACCESS reporting)

ATMOSPHERICSCIENCEDATA CENTER Support Plans and Issues

04/19/2312

• NASA-LaRC and NCDC – work through support plan items in later sessions

• Schedule of Tasks and Performers• Communication approach within team (internal and

ahead of required ACCESS reporting)• SAA Approach

ATMOSPHERICSCIENCEDATA CENTER Support Plans and Issues

04/19/2313

ATMOSPHERICSCIENCEDATA CENTER

MIIC-2 Project Wiki Status Pages – Ready for Members to Join in!

04/19/2314

ATMOSPHERICSCIENCEDATA CENTER

Background

04/19/2315

• NFS path over 1 GigE for now from MIIC-2 through OPeNDAP to ASDC Data holdings on the 1 PB DPO (CALIPSO and CERES, captive 1 month each of VIIRS and GOES data)

• Hyrax 1.9 with MIIC-2 plugins is installed on datasrvr2 (latest and greatest version from the OpENDAP group for custom MIIC services) datasrvr2 is an IBM x3650M2 with two Intel Xeon E5-2650 CPUs. The CPUs are 8-core 16 thread CPUs running at 2.0Ghz, it has 32GB of memory. It is running Redhat Enterprise Linux 6.3, it currently has a dual port 10g card installed (not configured, but has 2 10GigE GBICs), and 4 1GigE ethernet ports, three of which are configured. The server is located on ROW I, RACK D, at rack position 26.

• MIIC-2 services in Tomcat container/Apache with UI web page that comes up on client browser and also RESTful API for m2m batch – currently on desktop in Eclipse environment! PostgreSQL DB is on desktop so actual performance is skewed until more realistic configuration is achieved. Things are moving on to datasrvr2.

ATMOSPHERICSCIENCEDATA CENTER ASDC MIIC-1 Configuration

04/19/2316

• Development Environment: Aron installing an Eclipse Metrics plugin and will present its findings. A bug in Metrics necessitates a temporary downgrade to Java 1.6.

• CM tools for now (svn and TRAC)and planned (repository will be Atlassian stash/git)

• Eclipse + Spring tool suite. Migrating to Atlassian Project Mgtmt suite: Confluence, Jira, Bamboo, Stash (ASDC uses Atlassian)

• Testing: Some JUnit tests for unit test and higher-level end-to-end integration test. Currently do not have any coverage type tests.(use Bamboo)

• For the OPeNDAP plugin, use valgrind + memcheck to test for C++ memory leaks. (C++ is used for the OPeNDAP plug-in modules)

• Performance Testing: None currently on the Java side, as our current performance testing to date has identified the biggest performance blocks at the server. On OPeNDAP we use valgrind + massif to quantify memory allocations. (look into AppDynamics)

ATMOSPHERICSCIENCEDATA CENTER ASDC MIIC-1 MIIC-2 S/W Environment

04/19/2317

• TLE provider: Two-Line Ephemeris data are currently accessed from angler and incurs 35-second latency. Need to move information into PostgreSQL DB within MIIC-2 environment and connect to Celestrack (sp?) to get daily updates.

• Database: PostgreSQL, schema is auto-generated by Hibernate annotated Java. Base size is currently trivial (7026k). Need DB size of a one month study. ASDC uses Irwin for schemas.

ATMOSPHERICSCIENCEDATA CENTER ASDC MIIC-1 MIIC-2 S/W Environment

04/19/2318

• Just a template!!! – Demonstrate that ….

• Functional and V&V Testing completed• Infrastructure reviews completed

– Obtain NASA ASDC Management approval to proceed with running ACS-CALIPSO

• Actual schedule for production run to be determined

ATMOSPHERICSCIENCEDATA CENTER Software Packages Status