Smart InitializationSmart Initialization
Mike Romberg
Sept. 23-27, 2002 Smart Initialization 2
LectureLecture
Smart Init: what, why, how? Configurability Structure of Smart Init Modules Override Procedures Examples of Algorithms Enhancements Laboratory
Sept. 23-27, 2002 Smart Initialization 3
What is it?What is it?
Derives sensible weather elements from model data. Creates the IFP databases from the D2D data.
Works on models, not MOS
Sept. 23-27, 2002 Smart Initialization 4
Why is it needed?Why is it needed?
Models do not provide sufficient spatial resolution.
Models do not provide set of needed sensible weather elements.
Sept. 23-27, 2002 Smart Initialization 5
Derived Surface TDerived Surface T
Raw modelData at 80km
2 meter Temperature
Topography adjustments made
Surface Temperature
Sampled to 5kmresolution
Sept. 23-27, 2002 Smart Initialization 6
Characteristics of Smart InitCharacteristics of Smart Init
Field tailorable algorithms Can add new weather elements Can add local models, if the netCDF files
are in the correct D2D-style format Dependencies automatic Grids are generated as D2D data arrives
Sept. 23-27, 2002 Smart Initialization 7
How it worksHow it works
ifpServer
IFPGrid
Databases
Stores generated grids
Destination Database
4Determines what
is needed to generate, gets/stores data
3
Smart init module
Starts appropriate module
2
D2DnetCDFModel
Databases
ifpserver monitorsfor data arrival
SourceDatabase
1
Sept. 23-27, 2002 Smart Initialization 8
Configurability Configurability (serverConfig/localConfig)(serverConfig/localConfig)
INITMODULES Maps algorithms modules to models INITMODULES = { “MesoEta” : [“MESOETAU”, “MESOETAS”], “Eta” : [“ETA”], “LAPS” : [“LAPS”] }
INITMODULES = { “smart init module” : [‘D2D Source Databases’],
}
Two sources
Different names
Sept. 23-27, 2002 Smart Initialization 9
Configurability Configurability (serverConfig/localConfig)(serverConfig/localConfig)
INITSKIPS Can skip certain model runs (INITSKIPS)
INITSKIPS = {“RUC” : [1,2,4,5,7,8,10,11,13,14,16,17,19,20,22,23]}
INITSKIPS = {“D2D ModelName” : [hour1,hour2,hour3 to skip]
Sept. 23-27, 2002 Smart Initialization 10
Configurability Configurability (serverConfig/localConfig)(serverConfig/localConfig)
Number of D2D Versions Not directly associated with smart init Does NOT override D2D purging. Default is 2 versions.
D2DDBVERSIONS = { “ETA”: 3, “NGM”: 1 }
D2DDBVERSIONS = { “D2D model name” : numOfVersions, }
Sept. 23-27, 2002 Smart Initialization 11
Configurability Configurability (serverConfig/localConfig)(serverConfig/localConfig)
Smart init modules Can override and provide own Coded in Numerical Python One file per new model or changed model
Sept. 23-27, 2002 Smart Initialization 12
Smart Init structureSmart Init structure
File locations etc/BASE, etc/SITE Override files through etc/SITE Uses Python inheritance
Forecaster(etc/BASE/Init.py)
EtaForecaster(/etc/BASE/Eta.py)
serverConfig/localConfigINITMODULES
import Forecaster
MyEtaForecaster(/etc/SITE/MyEta.py)
import EtaForecaster
tells ifpServer whatto run
Sept. 23-27, 2002 Smart Initialization 13
Concept of Source/Destination Concept of Source/Destination Database – Database – (needed in defining new smart init modules)(needed in defining new smart init modules)
Source database – D2D netCDF file Destination database – IFP
ifpServer
Smart Init
D2D Database/data/fxa/…
AWIPS D2D
Source
IFP Database(ifpServer)
Destination
Sept. 23-27, 2002 Smart Initialization 14
Format of Smart Init FileFormat of Smart Init File
Class Statement Levels (function) Calc Functions
def calcT(arguments): Helper Functions (if needed) Main Declaration
Sept. 23-27, 2002 Smart Initialization 15
Smart Init InheritanceSmart Init Inheritance
Forecaster (base class)Init module
EtaForecasterEta module
MRFForecasterMRF module
NGMForecasterNGM module
AVNForecasterAVN module
Sept. 23-27, 2002 Smart Initialization 16
Format of smart init fileFormat of smart init file
from Init import *class EtaForecaster(Forecaster): def __init__(self): Forecaster.__init__(self, "ETA", "Eta")
def levels(self): return ["MB1000", "MB950", “MB900", "MB850", "MB800", "MB750", "MB700","MB650","MB600“]
Class Statement and LevelsImport from next class
Constructor for parent
Available levelsETA is D2D source, Eta is IFP destination
Sept. 23-27, 2002 Smart Initialization 17
Format of smart init fileFormat of smart init file
def calcSnowAmt(self, T, FzLevel, QPF, topo):
m1 = less(T, 9) m2 = greater_equal(T, 30) snowr = T * -0.5 + 22.5 snowr = where(m1, 20, snowr) snowr = where(m2, 0, snowr) snowamt = where(less_equal(FzLevel - 1000, topo * 3.048), snowr * QPF, 0) return snowamt
Calc FunctionsArguments
Returns the calculated grid
Sept. 23-27, 2002 Smart Initialization 18
Format of smart init fileFormat of smart init file
def linear(self, xmin, xmax, ymin, ymax, we): m = (ymax – ymin) / (xmax – xmin + 0.0000001) b = ymin – m * xmin return m * we + b
Example of a helper functionArguments
Returns the answer
Sept. 23-27, 2002 Smart Initialization 19
Format of smart init fileFormat of smart init file
def main(): EtaForecaster().run()
if __name__ == "__main__": main()
Main declarationNeeds to match class name
Sept. 23-27, 2002 Smart Initialization 20
calc Argumentscalc Arguments. . WARNING: These are different than smart tool argumentsWARNING: These are different than smart tool arguments
Weather element in destination databaseTparmName
Model topo in meters, from source database
stopostopo
High-res topo in meters, from destination database
topotopo
Cube of data, for the levels(), from source database
rh_cparmName_c
Single grid for ParmName/level – source database
t_FHAG2parmName_level
Sept. 23-27, 2002 Smart Initialization 21
calc Arguments calc Arguments (cont.)(cont.)
ctime Time from the source database grid currently being calculated, as a time range tuple (startTime, endTime) in seconds since epoch
mtime Time in the destination database grid currently being calculated, as a time range tuple (startTime, endTime), in seconds since epoch
stime Number of seconds from the model basetime currently being calculated. (Seconds since the model analysis time.)
Sept. 23-27, 2002 Smart Initialization 22
calc Argumentscalc Arguments
NOT the Fcst database as in smart tools Can use any combination of
source/destination weather elements in the calcXXX functions.
At present time, cannot easily have multiple input source databases
Sept. 23-27, 2002 Smart Initialization 23
Overriding AlgorithmsOverriding Algorithms
Why override? I don’t like existing algorithms. I need additional sensible wx elements.
Forecaster EtaForecaster
def calcSnowAmt() def calcSnowAmt()
MyEtaForecaster
We will redefine the equation for SnowAmt
Sept. 23-27, 2002 Smart Initialization 24
Steps to OverrideSteps to Override
Create new smart initialization module. If you modify the ones in etc/BASE they will go
away when you upgrade! Test out new smart initialization modules. Put it into routine operations.
Sept. 23-27, 2002 Smart Initialization 25
Format of Override FileFormat of Override File
Similar to existing smart init modules. Subtle differences, mainly due to
inheritance.
Sept. 23-27, 2002 Smart Initialization 26
Format of override fileFormat of override file
from Eta import *class MyEtaForecaster(EtaForecaster): def __init__(self): EtaForecaster.__init__(self)
def calcSnowAmt(self, T, QPF): m2 = less_equal(T, 32) snowamt = where(m2, 10.0*QPF, 0) return snowamt
def main(): MyEtaForecaster().run()
if __name__ == “—main__”: main()
Override Eta
Override function
Needs to match class name
New class name, differentinheritence structure
Sept. 23-27, 2002 Smart Initialization 27
Testing your new moduleTesting your new module
Can run from command line ifpInit –t modeltime algFile ifpInit –t modeltime –a algFile Model time in yyyymmdd_hhmm format. -a switch forces all grids to be generated
ifpInit –t 20011116_1200 –a MyEta
Sept. 23-27, 2002 Smart Initialization 28
Testing your new moduleTesting your new module
Run from the command line to see progress and problems.
Check the log files for errors …/data/logfiles/dateDirectory/EtaInit
Check the data in the GFE.
Sept. 23-27, 2002 Smart Initialization 29
Okay, it works, now what?Okay, it works, now what?
Put it into operational use by: Redefining INITMODULES in localConfig
serverConfig.INITMODULES[“MyEta”] = [“ETA”]
del serverConfig.INITMODULES[“Eta”]
Don’t forget to disable the original module!
Sept. 23-27, 2002 Smart Initialization 30
Adding New AlgorithmsAdding New Algorithms Define new algorithm module In localConfig, add new weather elements
to destination database(s) before testing. Test Modify localConfig’s INITMODULES.
Sept. 23-27, 2002 Smart Initialization 31
Adding Local ModelsAdding Local Models netCDF files must have certain information in order
for GFESuite to recognize them If displayable on d2d, probably close to ok. Need geographic information in netCDF.
Inherit from Forecaster, look at Eta.py for example. Similar procedure: write module, test, modify
INITMODULES. May need to modify D2DDIRS. Don’t forget - you can see these models on GFESuite
without smart init!
Sept. 23-27, 2002 Smart Initialization 32
Useful functions in Init.pyUseful functions in Init.py
def linear(self, xmin, xmax, ymin, ymax, we): -- linear interpolation
def FtoK(self, t): -- convert F to Kelvin
def KtoF(self, t): -- convert Kelvin to Farhenheit
def esat(self, temp): -- saturation vapor pressure
def self._empty –- returns grid of all zeros
def self.levels() –- returns list of levels (MB500, MB450,)
def self._minus –- return grid of all –1s
def self.pres –- return list of levels as numbers (500, 450,)
Sept. 23-27, 2002 Smart Initialization 33
Accessing Data TypesAccessing Data Types
Scalar T, simply a numerical grid
Vector tuple, V[0] is magnitude, V[1] is direction
Weather tuple, W[0] is grid, W[1] is key Key is set of Weather ugly strings Grid value indexed into key gives real value
Smart Tool lecture/lab covered specifics of types.
Sept. 23-27, 2002 Smart Initialization 34
GotchasGotchas Conditional statements
Unlike regular Python, both the true and the false are always executed in “where” statements
Standard Python if x != 0:
y = z / xelse: y = 0.0
Numerical Python where(not_equal(x, 0.0), z/x, 0.0))
where(not_equal(x, 0.0), z/(x+0.00001), 0.00)
Sept. 23-27, 2002 Smart Initialization 35
Future EnhancementsFuture Enhancements
Ability to access multiple sources Useful for model blending
Syntax identical for numerical smart tools and smart initialization Fairly close now.
Sept. 23-27, 2002 Smart Initialization 36
Examples of Algorithms - skyExamples of Algorithms - sky def calcSky(self, gh_c, rh_c, topo): # only use the first levels (up to MB600) gh_c = gh_c[:9,:,:] rh_c = rh_c[:9,:,:] rh_c = where(less(rh_c, 50), 0, rh_c) m1 = logical_and(greater(rh_c, 50), less(rh_c, 70)) m2 = logical_and(greater(rh_c, 70), less(rh_c, 85)) m3 = greater(rh_c, 85) skylayer = where(m1, self.linear(50, 70, 0, 25, rh_c), where(m2, self.linear(70, 100, 25, 100, rh_c), where(m3, 100, rh_c)))
Trim down levels
Rh<50%, then 0
Create masks,50-70%, 70-85%,> 85% Calc sky cover at each level
Sept. 23-27, 2002 Smart Initialization 37
Examples of Algorithms - skyExamples of Algorithms - sky skylayer = skylayer / 100 sky = skylayer[0] for i in xrange(1, skylayer.shape[0]): sky = sky + skylayer[i] - sky * skylayer[i]
sky = clip(sky, 0, 1) sky = sky * 100 return sky
Set to 0->1, from 0->100
Start at 1st level
Sum up the column
So…what is wrong with this algorithm?
Sept. 23-27, 2002 Smart Initialization 38
Laboratory ExercisesLaboratory Exercises
Override the Eta algorithm for T Simply use 2m FHAG directly Enhance to use dry adiabatic rate for topo
corrections Add a new model (UKMET)
Add UKMET to localConfig Calculate QPF, Wind, and T for UKMET