Welcome to the NOAA Institutional Repository | Evaluation of an automated acoustic beaked whale detection algorithm using multiple validation and assessment methods - :4462 | National Marine Fisheries Service (NMFS)
Stacks Logo
Advanced Search
Select up to three search categories and corresponding keywords using the fields to the right. Refer to the Help section for more detailed instructions.
Clear All Simple Search
Advanced Search
Evaluation of an automated acoustic beaked whale detection algorithm using multiple validation and assessment methods
  • Published Date:
Filetype[PDF - 1.32 MB]

  • Corporate Authors:
    Southwest Fisheries Science Center (U.S.)
  • Series:
    NOAA technical memorandum NMFS
  • Document Type:
  • Description:
    "Currently, the acoustic detection of beaked whales during passive acoustic surveys requires trained acousticians to identify beaked whale signals with the aid of various software programs. The development of reliable automated detection and classification methods will enable passive acoustic approaches to better meet monitoring needs for real-time mitigation of industry and military impacts. During ongoing development of automated beaked whale detectors and classifiers it will be important for researchers at different institutions to utilize standardized metrics of performance. At the Southwest Fisheries Science Center (SWFSC), automated detection algorithms for Cuvier's beaked whale (Ziphius cavirostris) and Baird's beaked whale (Berardius bairdii) were developed using PAMGUARD software (Douglas Gillespie: www.pamguard.org). To evaluate the performance of these beaked whale detectors, 15 ten-minute recording segments were processed in PAMGUARD, and the resulting signal detections were compared to manual logs of beaked whale signals confirmed by an experienced acoustician. The comparison was conducted using three methods: precise timestamp matching between manual and automated detections, detection counts from one-minute time bins, and binary presence/absence detection classification of one-minute bins. The detections were scored as true positive, false positive, false negative or false classification. Detector efficacy was quantified using measures developed for information retrieval systems (precision, recall, and F-score) as well as the Receiver Operating Characteristic. Calculated performance scores were compared across evaluation methods. We found that the method used to evaluate detector functionality greatly influences the resulting performance scores and subsequently our perception of detector ability. Therefore, it will be important for researchers to clearly communicate methods and results of detector evaluation. To allow for greatest precision and applicability to different recording datasets, we recommend that beaked whale detectors be evaluated using timestamp matching between manual and automated detections in trial datasets and that F-scores be used to compare detectors. This approach avoids problems associated with binning datasets by eliminating the need for a measure of false negatives"--Abstract.

  • Supporting Files:
    No Additional Files