Mining User Forums to Evaluate Quality-in-Use of Environmental Software

Abstract

Environmental research software applications are foundational tools for modeling and predicting Earth system dynamics. Among these, reactive transport models (RTMs) are particularly important because they simulate critical system processes that affect contaminant transport and water quality. Many RTMs are developed by domain scientists rather than software engineers, and have quality-in-use (QIU) issues that frustrate end users. Addressing these frustrations requires a clear understanding of the problems and challenges that users experience. We characterize the QIU frustrations most concerning to RTM users using a systematic approach. Specifically, we analyze RTM user forums and codify user-reported challenges by operationalizing the QIU characteristics and subcharacteristics defined in the ISO/IEC 25019:2023 standard. We extracted 3,941 forum threads from four widely used RTM user forums. Terms and phrases related to QIU barriers were identified using natural language processing (NLP) and mapped to QIU subcharacteristics to calculate their frequencies. The QIU characteristic of Beneficialness challenged users most frequently. Within Beneficialness, Usability caused the most frustration across the four forums, suggesting users struggle to achieve their goals effectively, efficiently, and with satisfaction. These findings highlight the importance of providing clear, accurate user resources, such as documentation and tutorials, to support research goals. By operationalizing the full scope of the QIU standard, we offer a transferable method for identifying end-user challenges across domains. By treating QIU as a first-class citizen, developers can improve user satisfaction and develop user-centric, trustworthy software systems across software domains.