User centered approach of applicability domain analysis for improving AI-assisted decision-making

AI-assisted decision-making is widely used in the chemical domain for accelerating discovery of materials. There is an increase in the proliferation of foundation models as the underlying technology of such AI-powered tools. The Organization for Economic Co-operation and Development (OECD) Validation Principle 3 expresses the need to "establish the scope and limitations of a model based on the structural, physicochemical and response information in the model training set". This analysis of applicability domain (AD) is an important concept in the context of safety assessment for fields like toxicology. There is a wide range of methods available for the study of AD based on varying definition of the AD concept. These methods often take into account different types of information. The hypothesis driving every AD study is different and model developers need to take this into account while providing experts with the means to understand the prediction and the decision-making steps. And hence we believe that applicability domain analysis of models should be user-driven. Our approach starts out by assessing the information needs of the expert end-user who will be using such models in their decision-making process. We explore this idea using a foundation-model specialized for chemical representation and by characterizing its uncertainty in predicting toxicity of PFAS through the use of mathematical and domain-specific metrics. We use semi-structured interviews with experts in the field of chemistry to systematically elicit user needs. In doing so, we focus the uncertainty analysis towards information that the user considers relevant and meaningful. Results demonstrate the level of alignment, or the lack thereof, between user's information needs and uncertainty information considered relevant by model developers. While the results highlight the importance of appropriate information for engendering trust in the model and its outcomes, we recommend user involvement from the outset of the modeling process.