Automating metamorphic testing by predicting metamorphic relations for complex and vulnerability-prone applications

Abstract

Software plays an important role in our daily operations and is a critical component of our infrastructure. Testing is necessary to evaluate the effectiveness of any software. To accomplish this, there is a fundamental need for an oracle, which determines whether the observed behavior of the software accurately reflects its intended functionality. However, software testing can pose significant challenges due to its complexity and the necessity of having a reliable oracle, often referred to as the oracle problem in software testing. Metamorphic Testing (MT) can alleviate the oracle problem because it focuses on evaluating software based on its inherent characteristics or properties. MT is a sophisticated technique that involves generating a variety of inputs for a program, subjecting them to predefined transformations, and subsequently comparing the resulting outputs with the original ones to verify the correctness of the program's behavior. Metamorphic Relations (MRs) are central to MT because they establish the relationships between the inputs and outputs of the system being tested and specify how they should change when the inputs are altered. Typically, identifying MRs is a manual process that often necessitates collaboration with domain experts, especially when testing complicated programs. Consequently, this task can be labor-intensive and prone to errors. Therefore, the development of automated methods for identifying MRs holds the potential to enhance the efficiency and effectiveness of MT, making it a more practical and reliable approach for ensuring the reliability of complex software systems. Hence, I employ MT techniques to analyze the software's behavior and anticipate and define MRs to achieve this goal. By predicting MRs, I streamline the testing process significantly. This entails automating the assessment of software behavior, reducing the reliance on manual testing procedures. In this thesis, I use machine learning classification models to predict MRs using data from diverse fields to identify faults. This approach predicts MRs for more complicated programs, such as Matrix Calculation Programs. Next, I examine the feasibility of MRpredT, a Text Classification-Based Machine Learning approach to predict MRs using only their program documentation as input. Then, I study the scope of testing applications with security flaws using MT. A systematic mapping study that documents the latest empirical research in web application security vulnerability detection indicates that vulnerability testing also encounters the oracle problem due to the vast range of inputs. Afterward, I introduce new MRs through a case study to test banking functions and demonstrate an MT framework. Finally, I detected vulnerabilities using MT, which led me to build an automated approach for vulnerable programs in online banking applications. It offers a catalog of 8 system-agnostic MRs to automate security testing for detecting these vulnerabilities among the OWASP Top 10. All the study results demonstrate that these approaches are theoretical and practical. It scales effectively, allowing for overnight automated software testing, and positions MT as a valuable and powerful tool for enhancing the correctness of any software system application.