© Crown copyright Met Office Verification of forecasts of Cbs Bob Lunnon, Aviation Outcomes Manager, Met Office WAFS Science meeting, Washington, April.

© Crown copyright Met Office

Verification of forecasts of Cbs Bob Lunnon, Aviation Outcomes Manager, Met Office

WAFS Science meeting, Washington, April 21st 2009

© Crown copyright Met Office

Acknowledgement

Most of this work has been done by Andrew Mirza (some help from Lauren Reid)

Thanks also due for use of lightning data (Sferics), and for availability of US products

© Crown copyright Met Office

Summary of presentation

• Background on lightning detection system (Sferics) used to perform verification

• Results of verification

• Conclusions/Recommendations

© Crown copyright Met Office

Met Office Sferics system• System detects the vertical component of electromagnetic

radiation generated by a lightning discharge at around 13 kHz (VLF) received at a number of `sensor’ sites

• Strong EM emissions around 13 kHz

• Caused by rapid neutralisation of charge, in the lowest few hundred metres of C-G stroke

• Atmospheric attenuation minima close to 13 kHz

• The atmoSFERIC wave can propagate over thousands of kilometres along the earth / ionosphere waveguide

• 12 sensors in the current network connected to Exeter by communication links

• Received digitised waveforms sent to the Flash Location Processor (FLP) via a Data Collection System (DCS) located in a `fire-walled’ DMZ

• Location of a `flash’ or `fix’ determined by the Arrival Time Differences (hence ATD)

• Differences between at least 4 sensor out-stations are used to calculate lightning position by iterative method

© Crown copyright Met Office

Total number of strikes recorded in July 2007

© Crown copyright Met Office

Sferics – area of reliable data

• between 40 S and 80 N and between 110 W and 80 E

• Verification data generated in latitude bands (and crude longitude divisions)

• South of 20s

• 20 South to 20 North

• 20 North to 50 North

• North of 50 North

© Crown copyright Met Office

Cb verification – data to be presented• In the WAFS Science meeting in April in

Washington we presented results for November 08 and January 09

• We were asked to generate statistics for N Hemisphere summer, e.g. May 09-July09

• There were data availability problems for July 09 (data for the UK SIGWX charts are not available, only 60% of US GRIB data available)

• We present data for May 09, June 09 and July 09 today

• We have plots for November 08 and January 09 if people are interested

© Crown copyright Met Office

Southern Hemisphere extra-tropics, May 09

ROC CB - May 2009

0

0.1

0.2

0.3

0.4

0.5

0.6

0.7

0.8

0.9

1

0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1

False -alarm rate

Hit

rate

SF1 - UK - T+12

SF1 - US - T+12

SF1 - UK - T+24

SF1 - US - T+24

SF1 - UK - T+24

SF1 - US - T+24

© Crown copyright Met Office

Tropics, May 09

ROC CB - May 2009

0

0.1

0.2

0.3

0.4

0.5

0.6

0.7

0.8

0.9

1

0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1

False -alarm rate

Hit

rate

SF2 - UK - T+12

SF2 - US - T+12

SF2 - UK - T+24

SF2 - US - T+24

SF2 - UK - T+24

SF2 - US - T+24

© Crown copyright Met Office

Northern Hemisphere mid latitudes, May 09

ROC CB - May 2009

0

0.1

0.2

0.3

0.4

0.5

0.6

0.7

0.8

0.9

1

0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1

False -alarm rate

Hit

rate

SF3 - UK - T+12

SF3 - US - T+12

SF3 - UK - T+24

SF3 - US - T+24

SF3 - UK - T+24

SF3 - US - T+24

© Crown copyright Met Office

Northern Hemisphere, polar area, May 09

ROC CB - May 2009

0

0.1

0.2

0.3

0.4

0.5

0.6

0.7

0.8

0.9

1

0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1

False -alarm rate

Hit

rate

SF4 - UK - T+12

SF4 - US - T+12

SF4 - UK - T+24

SF4 - US - T+24

SF4 - UK - T+24

SF4 - US - T+24

© Crown copyright Met Office

All latitudes, May 09

ROC CB - May 2009

0

0.1

0.2

0.3

0.4

0.5

0.6

0.7

0.8

0.9

1

0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1

False-alarm rate

Hit

rate

SF5 - UK - T+12

SF5 - US - T+12

SF5 - UK - T+24

SF5 - US - T+24

SF5 - UK - T+24

SF5 - US - T+24

© Crown copyright Met Office

Southern Hemisphere extra-tropics, June 09

ROC CB - June 2009

0

0.1

0.2

0.3

0.4

0.5

0.6

0.7

0.8

0.9

1

0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1

False -alarm rate

Hit

rate

SF1 - UK - T+12

SF1 - US - T+12

SF1 - UK - T+24

SF1 - US - T+24

SF1 - UK - T+24

SF1 - US - T+24

© Crown copyright Met Office

Tropics, June 09

ROC CB - June 2009

0

0.1

0.2

0.3

0.4

0.5

0.6

0.7

0.8

0.9

1

0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1

False -alarm rate

Hit

rate

SF2 - UK - T+12

SF2 - US - T+12

SF2 - UK - T+24

SF2 - US - T+24

SF2 - UK - T+24

SF2 - US - T+24

© Crown copyright Met Office

Northern Hemisphere mid latitudes, June 09

ROC CB - June 2009

0

0.1

0.2

0.3

0.4

0.5

0.6

0.7

0.8

0.9

1

0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1

False -alarm rate

Hitr

ate

SF3 - UK - T+12

SF3 - US - T+12

SF3 - UK - T+24

SF3 - US - T+24

SF3 - UK - T+24

SF3 - US - T+24

© Crown copyright Met Office

Northern Hemisphere, polar area, June 09

ROC CB - June 2009

0

0.1

0.2

0.3

0.4

0.5

0.6

0.7

0.8

0.9

1

0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1

False-alarm rate

Hit

rate

SF4 - UK - T+12

SF4 - US - T+12

SF4 - UK - T+24

SF4 - US - T+24

SF4 - UK - T+24

SF4 - US - T+24

© Crown copyright Met Office

All latitudes, June 09

ROC CB - June 2009

0

0.1

0.2

0.3

0.4

0.5

0.6

0.7

0.8

0.9

1

0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1

False-alarm rate

Hit

rate

SF5 - UK - T+12

SF5 - US - T+12

SF5 - UK - T+24

SF5 - US - T+24

SF5 - UK - T+24

SF5 - US - T+24

© Crown copyright Met Office

Southern Hemisphere extra-tropics, July 09 (no UK SIGWX)

ROC CB - July 2009

0

0.1

0.2

0.3

0.4

0.5

0.6

0.7

0.8

0.9

1

0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1

False -alarm rate

Hit

rate

SF1 - UK - T+12

SF1 - US - T+12

SF1 - UK - T+24

SF1 - US - T+24

SF1 - UK - T+24

SF1 - US - T+24

© Crown copyright Met Office

Tropics, July 09 (no UK SIGWX)

ROC CB - July 2009

0

0.1

0.2

0.3

0.4

0.5

0.6

0.7

0.8

0.9

1

0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1

False-alarm rate

Hit

rate

SF2 - UK - T+12

SF2 - US - T+12

SF2 - UK - T+24

SF2 - US - T+24

SF2 - UK - T+24

SF2 - US - T+24

© Crown copyright Met Office

Northern Hemisphere mid latitudes, July 09 (no UK SIGWX)

ROC CB - July 2009

0

0.1

0.2

0.3

0.4

0.5

0.6

0.7

0.8

0.9

1

0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1

False -alarm rate

Hit

rate

SF3 - UK - T+12

SF3 - US - T+12

SF3 - UK - T+24

SF3 - US - T+24

SF3 - UK - T+24

SF3 - US - T+24

© Crown copyright Met Office

Northern Hemisphere, polar area, July 09 (no UK SIGWX)

ROC CB - July 2009

0

0.1

0.2

0.3

0.4

0.5

0.6

0.7

0.8

0.9

1

0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1

False-alarm rate

Hit

rate

SF4 - UK - T+12

SF4 - US - T+12

SF4 - UK - T+24

SF4 - US - T+24

SF4 - UK - T+24

SF4 - US - T+24

© Crown copyright Met Office

All latitudes, July 09 (no UK SIGWX)

ROC CB - July 2009

0

0.1

0.2

0.3

0.4

0.5

0.6

0.7

0.8

0.9

1

0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1

False-alarm rate

Hit

rate

SF5 - UK - T+12

SF5 - US - T+12

SF5 - UK - T+24

SF5 - US - T+24

SF5 - UK - T+24

SF5 - US - T+24

© Crown copyright Met Office

Conclusions

• Overall “winner” appears to be US manual product (ranked first for combined latitudes for all five months under consideration)

• However this ranking reflects the number of grid-points in each latitude zone

• Number of flights per year is a function of latitude, strongly peaking in 20-50 north

• UK Automated product is ranked first in this zone in Nov 08, Jan 09, and May 09

• US Automated product shows useful skill in the tropics

• There is limited data available in NH polar and SH extra-tropics

• Any overall results are very sensitive to exact details of verification scheme

• Further work was addressed in earlier presentation

© Crown copyright Met Office

Questions & answers

© Crown copyright Met Office

Locally observed lightning strikes in S America, case in August 06

© Crown copyright Met Office

Data from UK sferics system for same day

© Crown copyright Met Office

Severe US storms – 13th March 2006

© Crown copyright Met Office

Results

• ROC curves for automated and manual products, two WAFCs, November 08 and January 09

© Crown copyright Met Office

Southern Hemisphere extra-tropics, November 08

ROC CB - November 2008 (Arithmetic Mean)

0

0.1

0.2

0.3

0.4

0.5

0.6

0.7

0.8

0.9

1

0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1

False -alarm rate

Hit

rate

SF1 - UK - T+12

SF1 - US - T+12

SF1 - UK - T+24

SF1 - US - T+24

SF1 - UK - T+24

SF1 - US - T+24

© Crown copyright Met Office

Tropics, November 08ROC CB - November 2008

(Arithmetic Mean)

0

0.1

0.2

0.3

0.4

0.5

0.6

0.7

0.8

0.9

1

0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1

False -alarm rate

Hitra

te

SF2 - UK - T+12

SF2 - US - T+12

SF2 - UK - T+24

SF2 - US - T+24

SF2 - UK - T+24

SF2 - US - T+24

© Crown copyright Met Office

Northern Hemisphere mid-latitudes, November 08

ROC CB - November 2008 (Arithmetic Mean)

0

0.1

0.2

0.3

0.4

0.5

0.6

0.7

0.8

0.9

1

0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1

False-alarm rate

Hit

rate

SF3 - UK - T+12

SF3 - US - T+12

SF3 - UK - T+24

SF3 - US - T+24

SF3 - UK - T+24

SF3 - US - T+24

© Crown copyright Met Office

Northern Hemisphere, polar area, November 08

ROC CB - November 2008 (Arithmetic Mean)

0

0.1

0.2

0.3

0.4

0.5

0.6

0.7

0.8

0.9

1

0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1

False-alarm rate

Hitr

ate

SF4 - UK - T+12

SF4 - US - T+12

SF4 - UK - T+24

SF4 - US - T+24

SF4 - UK - T+24

SF4 - US - T+24

© Crown copyright Met Office

All latitudes, November 08ROC CB - Novem ber 2008

(Arithm etic Mean)

0

0.1

0.2

0.3

0.4

0.5

0.6

0.7

0.8

0.9

1

0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1

False- alarm rate

SF5 - UK - T+12

SF5 - US - T+12

SF5 - UK - T+24

SF5 - US - T+24

SF5 - UK - T+24

SF5 - US - T+24

© Crown copyright Met Office

Southern Hemisphere extra-tropics, January 09

ROC CB - January 2009 (Arithmetic Mean)

0

0.1

0.2

0.3

0.4

0.5

0.6

0.7

0.8

0.9

1

0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1

False -alarm rate

Hit

rate

SF1 - UK - T+12

SF1 - US - T+12

SF1 - UK - T+24

SF1 - US - T+24

SF1 - UK - T+24

SF1 - US - T+24

© Crown copyright Met Office

Tropics, January 09ROC CB - January 2009

(Arithmetic Mean)

0

0.1

0.2

0.3

0.4

0.5

0.6

0.7

0.8

0.9

1

0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1

False-alarm rate

Hit

rate

SF2 - UK - T+12

SF2 - US - T+12

SF2 - UK - T+24

SF2 - US - T+24

SF2 - UK - T+24

SF2 - US - T+24

© Crown copyright Met Office

Northern Hemisphere mid-latitudes, January 09

ROC CB - January 2009 (Arithmetic Mean)

0

0.1

0.2

0.3

0.4

0.5

0.6

0.7

0.8

0.9

1

0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1

False -alarm rate

Hit

rate

SF3 - UK - T+12

SF3 - US - T+12

SF3 - UK - T+24

SF3 - US - T+24

SF3 - UK - T+24

SF3 - US - T+24

© Crown copyright Met Office

Northern Hemisphere, polar regions, January 09

ROC CB - January 2009 (Arithmetic Mean)

0

0.1

0.2

0.3

0.4

0.5

0.6

0.7

0.8

0.9

1

0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1

False-alarm rate

Hit

rate

SF4 - UK - T+12

SF4 - US - T+12

SF4 - UK - T+24

SF4 - US - T+24

SF4 - UK - T+24

SF4 - US - T+24

© Crown copyright Met Office

All latitudes, January 09

ROC CB - January 2009 (Arithmetic Mean)

0

0.1

0.2

0.3

0.4

0.5

0.6

0.7

0.8

0.9

1

0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1

False -alarm rate

Hit

rate

SF5 - UK - T+12

SF5 - US - T+12

SF5 - UK - T+24

SF5 - US - T+24

SF5 - UK - T+24

SF5 - US - T+24