BROCKMANN CONSULT 1 20. – 22.01.2009 20. – 22.01.2009 Finnish Environment Institute, Finnish Environment Institute, Syke Syke Helsinki Helsinki Programming with Programming with BEAM BEAM Norman Fomferra Norman Fomferra Carsten Brockmann Carsten Brockmann
38
Embed
B ROCKMANN C ONSULT 1 20. – 22.01.2009 Finnish Environment Institute, Syke Helsinki Programming with BEAM Norman Fomferra Carsten Brockmann.
This document is posted to help you gain knowledge. Please leave a comment to let me know what you think about it! Share it to your friends and learn new things together.
Transcript
BROCKMANN CONSULT
1
20. – 22.01.200920. – 22.01.2009Finnish Environment Institute, SykeFinnish Environment Institute, Syke
HelsinkiHelsinki
Programming with Programming with
BEAMBEAMNorman FomferraNorman Fomferra
Carsten BrockmannCarsten Brockmann
BROCKMANN CONSULT
2
Day 1: IntroductionDay 1: Introduction
1. Software OverviewAn introduction to VISAT, its user interface elements and the BEAM command line tools. Includes a detailed presentation of the BEAM standard I/O format.
2. Architecture and APIsA high level introduction into the BEAM 4.x architecture including the diverse application programming interfaces.
3. Runtime ConfigurationA detailed presentation of the BEAM installation directory structure, the standard modules, the 3rd party libraries used, the main entry points into the BEAM code.
4. Development and Build ProcessAn introduction to Java development with BEAM. The goal is to compile and run a simple program that uses the BEAM API with your preferred Java IDE.
Programming BEAM, Syke, 20.-22.01.2010
BROCKMANN CONSULT
3
Day 2: Using the BEAM APIDay 2: Using the BEAM API
1. A simple Data ProcessorWe develop a simple data processor, which will ingest an AATSR Level 1 data product and write a trivial cloud mask. The processor will use a simple command line interface. We will then review the output in VISAT.
2. BEAM Graph Processing Framework (GPF)A detailed introduction to the new BEAM graph processing framework.
3. Developing a GPF OperatorWe convert the simple data processor into a GPF operator. The processor will use the standard command line interface provided by the GPF.
4. BEAM ScriptingAn introduction to the new Scripting API introduced in Java 1.6 and a discussion of its relevance for BEAM.
Programming BEAM, Syke, 20.-22.01.2010
BROCKMANN CONSULT
4
Day 3: Extending BEAMDay 3: Extending BEAM
1. BEAM Module DevelopmentAn introduction to BEAM modules (plug-ins), the module descriptor file and module activators.
2. The Maven Build SystemMaven is a highly configurable build system which allows us to quickly create a module for BEAM.
3. Creating a BEAM ModuleWe add a module descriptor to the code developed so far in order to create a versioned, deployable and updateable BEAM module. The GPF operator becomes part of BEAM.
4. Integration with VISATWe develop a simple graphical user interface for the GPF operator.We integrate the GPF operator and its GUI into VISAT by adding a command which allows the users to invoke the operator from the VISAT Tools menu.
Programming BEAM, Syke, 20.-22.01.2010
BROCKMANN CONSULT
5
Wish list Wish list of participantsof participants
• Writing plug-ins (like a simple algorithm) • Batch processing• Automatic retrieval of new satellite products
(wouldn't that be a nice feature) • Command-line interfacing, interfacing with
MATLAB• Automatically producing graphics or
georeferenced sub-products • Pixel value retrieval, coordinate-based retrieval
for a large sample set • Overlaying transect data from external sources
with beam-supported productsProgramming BEAM, Syke, 20.-22.01.2010
BROCKMANN CONSULT
6Programming BEAM, ESRIN, 22.-26.09.2008
Why develop with BEAM?Why develop with BEAM?
• Access EO data, metadata, no-data, flags
• Simple & effective programming models
• Reuse a rich software infrastructure• It is free, it is open source, its Java • Review and reuse source code• BEAM code is widely used, it is in use• BEAM is supported by ESA
BROCKMANN CONSULT
7Programming BEAM, ESRIN, 22.-26.09.2008
Development Use CasesDevelopment Use Cases1. Use it
Write batch-mode scripts Program stand-alone applications Develop web-services
2. Extend it Add EO data readers, writers Add EO data processing nodes Add map projections, DEMs Add VISAT actions, tool bars and tool
modules Application and plug-in BEAM modules ceres-*.jar beam-*.jar
Programming BEAM, Syke, 20.-22.01.2010
BROCKMANN CONSULT
18
Invocation of Command-line Invocation of Command-line ToolsTools
Invocation of gpt.bat via a dedicated launcher program (ceres-launcher.jar). All JARs found in BEAM’s lib and modules directories are put on the classpath:
• Create processing chains or even graphs composed of processing nodes. Processing nodes shall be reusable in other graphs, e.g. MERIS Cloud Screening be easily configurable in terms of processing parameters be easily configurable in terms of the algorithm
implementation• Processor implementation code shall abstract from
physical file format of EO data file I/O configuration & parameterisation, parameter value access
• Avoid I/O overhead between processing steps• Exploit multi-core CPU architectures• The framework shall be extendible, e.g. it shall recognise
new processing node implementations at runtime
Programming BEAM, Syke, 20.-22.01.2010
BROCKMANN CONSULT
25
BEAM GPFBEAM GPF
• Allows to construct directed, acyclic processing graphs• Implements the “pull-processing” paradigm
A processor is represented by its processing graph Product processing requests are translated into raster-
tile-requests propagated backwards through the graph Tile requests are automatically parallelised depending on
CPU cores• Simple data processing and programming model
For a particular node, clients (Java developers) implement how a raster tile is being computed
Parameter values are “injected” by the framework• Generated user interfaces: Command-line (gpt) and VISAT
GUI• Operators can be dynamically added to BEAM via plug-in
◄ computes target tiles for◄ provides source tiles for
initialize()computeTile(...)
CloudMaskOp
createOperator()
CloudMaskOpSpi
◄ computes target samples for◄ retrieves source samples from
◄ creates target◄ uses as source
◄ knows its
knows its ►
BROCKMANN CONSULT
29
GPF Architecture (2)GPF Architecture (2)
• Clients provide their compiled Operator Java code packed into JAR modules
• A JAR module publishes its operator services via metadata (Service Provider Interface, SPI) META-INF/services/org.esa.beam.framework.gpf.OperatorSpi
• Once GPF is started, e.g. by visat.exe or gpt.bat, it loads all operator services it finds on its (dynamic) classpath %BEAM_HOME%/modules/*.jar %BEAM_HOME%/lib/*.jar
If single target bands can be computed independently of each other, e.g. NDVI or algorithms which perform single band filtering:
public class NoiseRedOp extends Operator { public void initialize() throws OperatorException { // validate and process source products and parameter values // create, configure and set the target product
}
public void computeTile(Band targetBand, Tile targetTile, ProgressMonitor pm) throws OperatorException { // Obtain source tiles for used bands of source products
// Process samples of source tiles to samples of target tile // Set samples of single target tile }}
If single target bands cannot be computed independently of each other, e.g. model inversion algorithms based on neural networks:
public class NoiseRedOp extends Operator { public void initialize() throws OperatorException { // validate and process source products and parameter values // create, configure and set the target product
// Obtain source tiles for used bands of source products // Process samples of source tiles to samples of target tiles // Set samples of all given target tiles }}
Programming BEAM, Syke, 20.-22.01.2010
BROCKMANN CONSULT
33Programming BEAM, ESRIN, 22.-26.09.2008
How GPF “drives” Processing How GPF “drives” Processing NodesNodes
1. GPF knows about all registered Operators (Plug-in)2. GPF encounters a required node (GUI, CLI, XML graph)
1. The client‘s Operator class is found by its alias2. The SourceProduct, TargetProduct and Parameter
annotations are analysed3. The client‘s Operator object is created 4. Source products and parameter values are injected
3. GPF calls Operator.initialize(). In this method, the client provides code to1. validate and process source products and parameter
values2. create, configure and set the target product
4. GPF calls either Operator.computeTile() or Operator.computeTileStack()in case raster data is required
BROCKMANN CONSULT
34
gpt – The GPF Command-Line gpt – The GPF Command-Line ToolTool