PhD Oral Exam – Hamad Mohammed Binsalleeh

Abstract

Malicious domains host Command and Control servers that are used to instruct infected machines to perpetuate malicious activities such as sending spam, stealing credentials, and launching denial of service attacks. Both static and dynamic analysis of malware as well as monitoring Domain Name System (DNS) traffic provide valuable insight into such malicious activities and help security experts detect and protect against many cyber attacks.

Advanced crimeware toolkits were responsible for many recent cyber attacks. In order to understand the inner workings of such toolkits, we present a detailed reverse engineering analysis of the Zeus crimeware toolkit to unveil its underlying architecture and enable its mitigation. Our analysis allows us to provide a breakdown for the structure of the Zeus botnet network messages.

In the second part of this work, we develop a framework for analyzing dynamic analysis reports of malware samples. This framework can be used to extract valuable cyber intelligence from the analyzed malware. The obtained intelligence helps reveal more insight into different cyber attacks and uncovers abused domains as well as malicious infrastructure networks. Based on this framework, we develop a severity ranking system for domain names. The system leverages the interaction between domain names and malware samples to extract indicators for malicious behaviors or abuse actions. The system utilizes these behavioral features on a daily basis to produce severity or abuse scores for domain names.

Since our system assigns maliciousness scores that describe the level of abuse for each analyzed domain name, it can be considered as a complementary component to existing (binary) reputation systems, which produce long lists with no priorities.

We also developed a severity system for name servers based on passive DNS traffic. The system leverages the domain names that reside under the authority of name servers to extract indicators for malicious behaviors or abuse actions. It also utilizes these behavioral features on a daily basis to dynamically produce severity or abuse scores for name servers.

Finally, we present a system to characterize and detect the payload distribution channels within passive DNS traffic. Our system observes the DNS zone activities of access counts of each resource record type and determines payload distribution channels. Our experiments on near real-time passive DNS traffic demonstrate that our system can detect several resilient malicious payload distribution channels.