Speaker
Description
Scientific models developed for research are commonly evaluated using selected events, with their input parameters fine-tuned for the specific task. Operational deployment, however, requires continuous execution with pre-determined parameters, while tolerating incomplete or inconsistent inputs. They furthermore need the ability to recover from infrastructure failures. The requirements for a research-focused model run and an operational system, even if based on the same model, are different. We report lessons learned from operating an automated heliophysics modelling pipeline connecting real-time data retrieval with MHD simulations, postprocessing, and web publication of the results.
Operational experience showed that many significant failures arose not within the scientific models themselves, but in the surrounding operational infrastructure. Input data might be delayed, incomplete, or incompatible with assumptions. Additional interruptions were introduced by HPC frontend and network errors. Long-duration execution further exposed instabilities not observed or avoidable in selected research cases. At the same time, forecast usefulness required balancing model resolution and assumptions against runtime, queuing time, and publication latency.
These experiences demonstrate the need for both consistent and timely input data, and constant monitoring and checkpointing of the system. Operational performance should be evaluated not only on model output but also using metrics such as run completion rate, input data age, manual-intervention rate, recovery time, and product latency. We conclude that operationalizing a scientific model is not equivalent to automating its execution. It requires both computational resilience, product latency and scientific validity as coupled design requirements. Continuous operation also provides a systematic stress that exposes hidden model assumptions and can guide subsequent research and model development.