Theses and Dissertations at Montana State University (MSU)
Permanent URI for this communityhttps://scholarworks.montana.edu/handle/1/732
Browse
2 results
Search Results
Item An evaluation of graph representation of programs for malware detection and categorization using graph-based machine learning methods(Montana State University - Bozeman, College of Engineering, 2023) Pearsall, Reese Andersen; Chairperson, Graduate Committee: Clemente IzurietaWith both new and reused malware being used in cyberattacks everyday, there is a dire need for the ability to detect and categorize malware before damage can be done. Previous research has shown that graph-based machine learning algorithms can learn on graph representations of programs, such as a control flow graph, to better distinguish between malicious and benign programs, and detect malware. With many types of graph representations of programs, there has not been a comparison between these different graphs to see if one performs better than the rest. This thesis provides a comparison between different graph representations of programs for both malware detection and categorization using graph-based machine learning methods. Four different graphs are evaluated: control flow graph generated via disassembly, control flow graph generated via symbolic execution, function call graph, and data dependency graph. This thesis also describes a pipeline for creating a classifier for malware detection and categorization. Graphs are generated using the binary analysis tool angr, and their embeddings are calculated using the Graph2Vec graph embedding algorithm. The embeddings are plotted and clustered using K-means. A classifier is then built by assigning labels to clusters and the points within each cluster. We collected 2500 malicious executables and 2500 benign executables, and each of the four graph types is generated for each executable. Each is plugged into their own individual pipeline. A classifier for each of the four graph types is built, and classification metrics (e.g. F1 score) are calculated. The results show that control flow graphs generated from symbolic execution had the highest F1 score of the four different graph representations. Using the control flow graph generated from symbolic execution pipeline, the classifier was able to most accurately categorize trojan malware.Item A framework to assess bug-bounty platforms based on potential attack vectors(Montana State University - Bozeman, College of Engineering, 2022) McCartney, Susan Ann; Co-chairs, Graduate Committee: Clemente Izurieta and Mike WittieCorporate computer security is becoming increasingly important because the frequency and severity of cyberattacks on businesses is high and increasing. One way to improve the security of company software is for a company to hire a third party to identify and report vulnerabilities, blocks of code that can be exploited. A bug-bounty program incentivizes ethical hackers (herein, 'researchers') to find and fix vulnerabilities before they can be exploited. For this reason, bug-bounty programs have been increasing in popularity since their inception a decade ago. However, the increase in their use and popularity also increases the likelihood of the companies being targeted by malicious actors by using a bug-bounty programs as the medium. The literature review and investigation into the rules and requirements for bug-bounty platform revealed that though the bug-bounty programs can improve a vendor's security, the programs still contain a serious security flaw. The platforms are not required to scan reports for malware and there is no guidance requesting the vendors scan for malware. This means it is possible to perform a cyberattack using malware as a report attachment. Through data collection from 22 platforms, an observational case study, and analysis of different malware, I have created a tool to assist vendors in selecting the platform of best fit and characterize the possible attack surfaces presented from the file options allowed on the platform. The outcome from this research is evidence of the importance of understanding the malware files used as report attachments. However, more research is needed in the relationship between file extensions and malware in order to thoroughly comprehend the attack surface capabilities, and to understand the trade-offs between security and convenience.