R2R QA for CTD Profile Data - UNOLS

R2RQAforCTDProfileData

StatusUpdateReportCyndyChandler(WHOI)

RollingDecktoRepository(R2R)

November2016AnnualRVTECMeeOng

CTD–ConducOvity,Temperature,Depth

Sea-Bird911plushUp://www.seabird.com/sbe911plus-ctd (Plueddemann,WHOI;OOIKN222,2014)

CTDdataexample

slide3of10

conOnentalshelf.gov

DEPTH(frompressure)

CTDdatafiles

•  VesselresidentCTDunitswithaddiOonalopOonalsensors(dissolvedO2,PAR,transmissometer,fluorometer)

•  CTDdata(andsetup)filesets•  Setoffilesforeachcast(CTDdeployment)

•  ConOnuousprofiledata(downandup)•  ‘boUlefiles’:boUlefirerecords

slide4of23

R2RQAforCTDprofiledata•  TeamMembers

CarolinaNobre(WHOI,PhysicalOceanographyDepartment)CyndyChandler(WHOI,BCO-DMO,R2R)LauraStolp(WHOI,ShipboardScienOficServices)

•  Generalupdateonprojectstatus:–  BatchSeaBirdProcessing

•  GeneratesprocessedCTDfiles–  CTDQAscript(python)thatrunsonCTDfilesfromeachcruise

•  generatesR2RQAsummaryreport(XML)Runon978cruisedatasets(rawCTDdatathathadbeenbrokenoutfromcruisedistribuOons)

–  ProcessedCTDdataproductsgeneratedfor853(ofthe978)

R2RQAforCTDprofiledata

IniOalresultsofQAprocessing:•  About10%couldnotbeassessed

•  missingfiles,PARsensorissues

•  CTDdatafrom853cruisesprocessed•  ~40%ofthecruisesareallgreenlights•  ~30%ofthecruiseshaveatleastoneyellowlight•  ~30%ofthecruiseshaveatleastoneredlight

slide6of23

QA Dashboard – CTD Example (Results displayed internally; pending feedback from experts)

slide7of10

QATestResultsDisplay

slide8of10

QATestResultsDisplay

slide9of23

CTDQAWorkflow

RawFiles ProcessedFiles

QAReportSeabird

Processing

PRODUCT PRODUCT

QATests

•  SeabirdModules•  datcnv•  wildedit•  Celltm•  filter•  loopedit•  Binavg(2db)•  Split

•  ParametersExtracted–  Temperature–  ConducOvity–  Oxygen–  AlOmeter–  …

RawFiles->SeabirdProcessing->ProcessedCTD

n  RawFilesn  .hex/.datn  .con/.xmlconn  .hdrn  .bln  .NAVn  ,,,

n  .cnvFilesn  No.ofStaOonsn  BoUlesFiredn  Max/mincast

depthn  Sensor

ConfiguraOon

–  Cruisemetadata–  ValidityoftheGPSdata–  PresenceofallexpectedrawSeaBirdfiles–  Presence/validityofdate/Omeentries

–  Sensors:–  PresenceofRedundantSensors–  DataRanges(comparisonwithmanufacturer

specsforeachsensor)–  SensorListandCalibraOonDates

–  DataCoverage:–  NumberofstaOonsdetected–  DetecOonofboUlesfired–  Minimumpressuretest(castswentbelow5m)

QA–Metadata,SensorsDataCoverage

QAProcessSteps

Processhasevolvedoverseveralyears;informedbyfeedbackfromChrisPaver(NCEI),shipboardtechs,R2Rpartners,WHOIPOscienOstsanddatamanagers•  AbilitytohandledifferentversionsSeaBirdSeaSon•  Abilitytomodify/create.psafilesandinsertthemintobatchprocessing(SeaSavesonwaresetupforWindows)

•  Addedseabirdmodules(Wildedit,LoopEdit,BinAvg,Split)togenerateprocessedCTDprofiledataproduct(2decibar,bin-averaged,down/upcastfiles)

slide13of23

QAProcessSteps•  R2RNavprocessingvalidatestemporalbounds(start/end

dates)andfillsinspaOalbounds(WESNbox)•  CTDQAprocesschecksagainstthosespaOalandtemporal

cruisebounds,andidenOfiesoutliercastsintheXMLreport•  R2RbreakoutprocessreadstheCTDQAXMLreportand

updatesSQLdatabase(ifneeded),andresubmitsfilestoNCEIifnecessary(newversion)

•  InaddiOontoindividualtestresults,CTDQAreportinfoblockincludeswhichcasts/filesfailedtests

•  BoththeLevel0(originalraw)andLevel1(processed)willbesubmiUedtoNCEI(oneQAreportappliestoboth)

slide14of23

QAProcessStepsCreateR2RQualityAssessment(QA)RollupReport•  XMLformat,compliantwithR2Rv1.0schema•  BasicprovenanceinformaOon(cruise,filesetID)•  InfoResultsSummary:•  TotalRawFiles,TotalProcessedFiles

BoUlesFired(True/False)%ofCastswithBoUlesFiredNumberofSensorsDetectedSensorInfolist:SensorName,SensorID,CalibraOonDateCreatesuiteofsensor-specificplots

•  FileSetInfo:basiccruiseandinstrumentmetadata•  Filemanifest:filename,IDandchecksumvaluesCTDrawandprocesseddataplusQAfilessubmiUedtoNCEI

slide15of23

QAProcessSensorTests

•  Min/maxrange(sensormanufacturerspecs)•  Gap•  Spike•  ConstantValue•  GradientTest•  OutlierTest•  DualSensorPresenceTest(TempandCond)•  DualSensorDifferenceTest(TempandCond)

slide16of23

QAProcessResults

•  CTDQAreport(XML)•  Sensor-specificQAplots•  QAdashboarddisplay(accesstoQAresults)•  Product:“ProcessedSeabirdCTDfiles”

•  ConsistentCTDdataproductfromeverycruise•  Downanduptracedatafileforeachcast•  .psa(processingfile)•  ConReports(ASCIIversionwithfullinstrumentmetadata,includingallsensorinfo)

slide17of23

QAPlots

slide18of23

QAPlots

slide19of23

FutureEnhancements

•  ConOnueevaluaOngwhysomecruisesarefailingcertaintests(adjustprocess)

•  Reviewtestresultswithdomainexperts•  ExposetheCTDQAtestresultsonthepublicdashboardview

Hurdles(andtherehavebeenafew…)

•  Syntaxofusersupplieddata•  Example:**LaOtude:3136.32

•  MismatchedFileNames•  DeckTests(missingfiles,emptyfiles)

–  (missingfilesareredforexample)

•  PARsensorinconsistency(detailsnextslide)

PARSensorIssue

TheabilitytocollectPARdataisdeterminedbytwoseparate’switches’,oneonthedeckunitandtheotherinthe.confileIfthetwodon’tmatch(switchforPARdataisonintheconfilebutnotonthedeckbox,orviceversa)youendupwitheithernoPARdataorbaddata.

slide23of10

Thanks!

QUESTIONS?