In this article · 7 sections
Every lightning vendor has a headline number. On its own, a headline number can be made to look good by a forecast that is no use to you. The way to read it is to ask which question it answers and which one it ignores, and the only way to know that is to know the method.
Every forecast ends in one of four boxes
Forecast verification starts with a simple table. For a yes-or-no event such as "lightning in this cell this hour", every forecast and every outcome lands in one of four boxes. The definitions below are the standard ones set out by the WWRP/WCRP Joint Working Group on Forecast Verification Research.
| Attribute | Lightning happened | No lightning |
|---|---|---|
| Forecast said yes | Hit | False alarm |
| Forecast said no | Miss | Correct negative |
Lightning happened
- Forecast said yes
- Hit
- Forecast said no
- Miss
No lightning
- Forecast said yes
- False alarm
- Forecast said no
- Correct negative
Every score you will be shown is a ratio of some of these boxes and leaves the others out. That is why no single score is enough.
The hit rate, and how it can be flattered
Probability of detection, also called the hit rate, is hits divided by hits plus misses: of the events that happened, how many had been forecast. It is the score most safety buyers care about first, because a miss is a strike nobody was warned about.
It also has a well-known weakness. The verification literature points out that the hit rate ignores false alarms, so it can be pushed up simply by forecasting "yes" more often. A forecast that flagged every cell all day would detect every strike and be useless on a field.
Flash’s published lightning figure is this kind of score. On the accuracy method page, accuracy is hits divided by hits plus misses, where a hit is a cloud-to-ground strike inside a 1×1 km cell that Flash had flagged at Advisory or higher during the preceding hour. On that definition the one-hour figure is up to 99.6%. The method page says plainly why that is not the whole story: the false-alarm side is tracked beside it, because a forecast that flagged everything would score a perfect hit rate.
False-alarm ratio: the cost of crying wolf
False-alarm ratio is false alarms divided by all the "yes" forecasts: of the times the forecast said lightning, how often none came. It measures the thing the hit rate ignores, and it is the number that decides whether your trainers keep trusting the horn. It has the opposite blind spot: it ignores misses, so a forecast that rarely says yes can keep it low while missing storms.
Do not confuse it with the false-alarm rate, which sounds the same and is not. The false-alarm rate, or probability of false detection, divides false alarms by all the occasions when nothing happened. In lightning, where most cells are quiet most hours, that denominator is huge and the rate looks tiny. If a vendor quotes a false-alarm figure, ask which one it is.
On the Flash method, the false-alarm ratio is false alarms divided by flagged cell-windows, and a false alarm is a flagged cell-window with no cloud-to-ground strike inside it. The hit rate and the false-alarm ratio are reviewed together before a figure is published, and the false-alarm count is never folded into the accuracy figure.
First-strike detection: the strike that matters most
A storm cell can put many strikes into the same 1×1 km cell. Once the first has landed, everyone knows lightning is there, and the later strikes are easy to "forecast". A hit rate counted over every strike can therefore be carried by the easy ones.
The first strike is the hard one, and it is the one a safety policy exists for. In the Flash glossary, the first strike is the first cloud-to-ground strike of a storm cell inside a given 1×1 km cell. A detection network can only report it after the fact; a prediction is scored on whether it had flagged the cell first. When you compare vendors, ask whether their figure counts the first strike separately or lets the storm’s later strikes pad the total.
All-clear precision: ending the delay at the right time
Clearing a field is half the decision. The other half is calling everyone back. An all-clear that comes too early puts people outside while the storm is still active, and one that comes too late loses time to a storm that has moved on.
The Flash method defines a correct all-clear as one that is not followed by a cloud-to-ground strike within the return-to-play window your policy sets. All-clear precision is the share of all-clears that meet that test. It ties the score to your rule, not the vendor’s. Note what it does not change: the return-to-play clock itself is set by your policy and anchored to the last detected strike.
Why the window matters more than the headline
Every one of these scores depends on how a hit is defined, and two settings do most of the work: how large the area is, and how long the window is. Widen either and hits become easier. A strike "somewhere in the county today" is a far easier forecast than a strike in one 1×1 km cell within the hour, and the same percentage means very different things on the two.
That is why the Flash figure is always printed with its window and its ground truth: one hour, per 1×1 km cell, scored against an independent record of cloud-to-ground strikes, with intra-cloud lightning excluded from both the forecast and the score. Only the one-hour window is scored. The 6-hour outlook is planning guidance and is not part of the figure.
- What counts as a hit: what area, what window, what alert level?
- What is the ground truth? A score against the vendor’s own output is circular.
- Which false-alarm figure is it: the ratio, or the rate?
- Is the first strike counted on its own, or swamped by the strikes after it?
- Can you re-run it? The Flash method page explains how to request the validation dataset and recompute the figure.
Frequently asked
False alarms divided by all the forecasts that said the event would happen. It measures how often a warning is not followed by the event, and it ignores events that were missed.
No. The ratio divides false alarms by all yes forecasts. The rate divides them by all the occasions when nothing happened, which makes it look much smaller for a rare event like lightning in one cell.
Because it ignores false alarms. A forecast can raise its hit rate by flagging more cells more often, so the hit rate has to be read beside the false-alarm ratio.
It is Flash’s one-hour lightning figure: hits divided by hits plus misses, where a hit is a cloud-to-ground strike in a 1×1 km cell Flash had flagged at Advisory or higher during the preceding hour. False alarms are tracked beside it.