Echodataflow: Recipe-based Fisheries Acoustics Workflow Orchestration

Details

• Journal Title:
  Proceedings of the Python in Science Conference
• Personal Author:
  Staneva, Valentina ; Butala, Soham ; Setiawan, Landung (Don) ; Lee, Wu-Jung
• NOAA Program & Office:
  OAR (Oceanic and Atmospheric Research) ; Ocean Exploration Program
• Description:
  With the influx of large data from multiple instruments and experiments, scientists are wrangling complex data pipelines that are context-dependent and non-reproducible. We demonstrate how we leverage Prefect (Prefect, 2024), a modern orchestration framework, to facilitate fisheries acoustics data processing. We built a Python package Echodataflow (Echodataflow, 2024) which 1) allows users to specify workflows and their parameters through editing text “recipes” which provide transparency and reproducibility of the pipelines; 2) supports scaling of the workflows while abstracting the computational infrastructure; 3) provides monitoring and logging of the workflow progress. Under the hood, Echodataflow uses Prefect to execute the workflows while providing a domain-friendly interface to facilitate diverse fisheries acoustics use cases. We demonstrate the features through a typical ship survey data processing pipeline.
• Keywords:
  FY21 NOFO
• Source:
  Proceedings of the Python in Science Conference (2024)
• DOI:
  https://doi.org/10.25080/JXDK4427
• ISSN:
  2575-9752
• Format:
  pdf
• Publisher:
  SciPy
• Document Type:
  Journal Article
• Funding:
  Grant no. NA21OAR0110201-T1-01
• License:
  https://creativecommons.org/licenses/by/4.0/
• Rights Information:
  CC BY
• Compliance:
  Submitted
• Main Document Checksum:
  urn:sha-512:1939de6037e4b6f54725d9dbd1c616dca33f7521b0497470d51284611190fe8742213ca035093e1f76afcb1e79fd4185de6a37bedf9d1bc3259eef3c3a84519f
• Download URL:
  https://repository.library.noaa.gov/view/noaa/71896/noaa_71896_DS1.pdf
• File Type:
  [PDF - 3.19 MB ]