Metamorphic relations ranking for reducing testing cost in scientific software

Abstract

Lack of automated test oracles is a major challenge faced when testing scientific software. An oracle is a mechanism determine whether test results are correct according to the expected behavior of the program. Metamorphic Testing (MT) is a testing technique that can be used to test such applications. This approach checks relations among multiple inputs and outputs of the program instead of checking the correctness of individual test outputs. Theses relationships are called Metamorphic Relations (MRs) and their violations indicates faults in System Under Test (SUT). Programs have several MRs with different fault detection effectiveness. Thus order in which they apply determines the efficiency of the testing process. Therefore in this work we propose a strategy to prioritize MRs based on their potential fault finding ability. Our strategy uses mutation testing to create a prioratized order of MRs for a given program. We evaluated our proposed approach using machine learning libraries in Weka as well as open source mathematical programs; these results show that our strategy is effective in developing a prioratized order of MRs that maximizes early fault detection. Our results show that in 126 methods we can detect 50.1%-100% faults using 25% of the MRs compared to a random order only detect 1.13%-100%.