Speaker
Description
We have developed data pipelines using Prefect for Solar Orbiter SIS, LRO LAMP, IMAP CoDICE, and TRACERS ACI instruments.
Prefect is an orchestrator for doing data pipelines, typically in the AI/ML realm, and it turns out to a huge benefit to doing our data pipelines for space science giving us observability and metrics. Although it is not perfect, there are some reasons why it can be advantageous to teams developing a pipeline when starting a project. Even more interesting is the case when a pipeline is fully developed and one wants to add operations to the pipeline. This talk or poster will talk about both types: for IMAP and TRACERS, the pipelines were developed from scratch and for LRO and SOLO, they were migrated to this method. We will discuss the pros and cons of this approach compared to other software including regular script files and other orchestrators such as Dagster and Airflow.