Applications of Hypervisors in Security

Abstract

As malware continue to evolve, so do the countermeasures which attempt to fight them. A modern computer system typically has many security services installed on top of its operating system which include antivirus, application-control, IDS, firewall, and many more. Modern operating systems are a highly complex pieces of software which typically contains millions of lines of code. Furthermore, primarily due to endless hardware support, new code is regularly added, resulting in a security sink with an open drain. Most security services run on top of the operating system and, therefore, are subject to the security of the operating system and its applications. In case of a vulnerability, these services can be removed, thus rendering them them completely useless. This thesis proposes a thin hypervisor-based architecture for a system on top of which a variety of security services can be implemented. These services run in a secure, isolated environment. Furthermore, the proposed system can hide the presence of these security services. The proposed system architecture provides strong security guarantees. The thesis presents four common, heavily researched security problems and proposes four solutions, which are all based on the proposed architecture. The proposed solutions can compete, and even outperform current solutions, both in terms of security and performance. Keywords: trusted computing, virtualization, hypervisor, thin hypervisor, unauthorised execution, malware analysis, code encryption, memory forensics