Speaker
Description
We developed and evaluated an automated approach for classifying the use of heliophysics instrument data across the scientific literature using a large language model (LLM). The system classifies data usage instances identified by an upstream pipeline, where each instance represents the use of observations from a specific mission, instrument, and observation period within a scientific paper. Each usage is classified according to two dimensions: provenance, determining whether a paper presents its own analysis of the data or reports results primarily derived from another work, and depth, determining whether the data are central to the scientific analysis or included only for illustration or context.
The classifier was evaluated on 480 data usage instances from 72 heliophysics papers selected to represent complex and ambiguous cases of data usage. The LLM was provided only with extracted quotations and contextual information rather than full manuscripts, and results were compared with classifications obtained from full-paper analysis.
Using extracted context alone, the classifier correctly identified both usage dimensions for 84% of cases. Providing full manuscripts increased accuracy to 90% but required substantially greater computational cost and introduced additional errors. Remaining misclassifications were primarily associated with limitations in extracted information rather than the LLM's classification ability.
These results demonstrate the potential for LLM-based methods to enable scalable characterization of instrument data usage across the heliophysics literature.