An Objective Scoring Method for Evaluating the Comparative Performance of Automated Storm Identification and Tracking Algorithms

Satrio, Clarice N.; Calhoun, Kristin M.; Campbell, P. Adrian; Steeves, Rebecca; Smith, Travis M.

doi:10.1175/WAF-D-22-0047.1

i

An Objective Scoring Method for Evaluating the Comparative Performance of Automated Storm Identification and Tracking Algorithms

2022
By Satrio, Clarice N. ; Calhoun, Kristin M. ; Campbell, P. Adrian ; ...

Details

Journal Title:

Weather and Forecasting
Personal Author:

Satrio, Clarice N. ; Calhoun, Kristin M. ; Campbell, P. Adrian ; Steeves, Rebecca ; Smith, Travis M.
NOAA Program & Office:

CIWRO (Cooperative Institute for Severe and High-Impact Weather Research and Operations) ; OAR (Oceanic and Atmospheric Research) ; NSSL (National Severe Storms Laboratory)
Description:

While storm identification and tracking algorithms are used both operationally and in research, there exists no single standard technique to objectively determine performance of such algorithms. Thus, a comparative skill score is developed herein which consists of four parameters, three of which constitute the quantification of storm attributes – size consistency, linearity of tracks, and mean track duration – and the fourth which correlates performance to an optimal post-event reanalysis. The skill score is a cumulative sum of each of the parameters normalized from zero to one amongst the compared algorithms, such that a maximum skill score of four can be obtained. The skill score is intended to favor algorithms which are efficient at severe storm detection, i.e., high-scoring algorithms should detect storms that have higher current or future severe threat and minimize detection of weaker, short-lived storms with low severe potential. The skill score is shown to be capable of successfully ranking a large number of algorithms, both between varying settings within the same base algorithm and between distinct base algorithms. Through a comparison with manually-created user datasets, high-scoring algorithms are verified to match well with hand analyses, demonstrating appropriate calibration of skill score parameters.
Keywords:

Algorithms Storms Weather Forecasting Weather Forecasting Atmospheric Science
Source:

Weather and Forecasting (published online ahead of print 2022)
DOI:

https://doi.org/10.1175/WAF-D-22-0047.1
Document Type:

Journal Article
Funding:

NA21OAR4320204 ; NA18OAR4590386
Rights Information:

Other
Compliance:

Submitted
Main Document Checksum:

urn:sha-512:2dfa2394f07d7acccefaf5c1d79e2f96a483c96512582d37a6aa2854aaf69e4e89a4895f48d209a218307a10de17ca030328aba6831ab0a1b9ae977ed7f30cf6
Download URL:

https://repository.library.noaa.gov/view/noaa/45579/noaa_45579_DS1.pdf
File Type:

[PDF - 6.92 MB ]

ON THIS PAGE

Details Related Documents

The NOAA IR serves as an archival repository of NOAA-published products including scientific findings, journal articles, guidelines, recommendations, or other information authored or co-authored by NOAA or funded partners. As a repository, the NOAA IR retains documents in their original published format to ensure public access to scientific information.