Preventing reverse engineering of native and managed programs

One of the important aspects of protecting software from attack, theft of algorithms, or illegal software use is eliminating the possibility of performing reverse engineering. One common method used to deal with these issues is code obfuscation. However, it is proven to be ineffective. Code encryption is a much more effective means of defying reverse engineering, but it requires managing a cryptographic key available to none but the permissible users. The thesis presents a system for managing cryptographic keys in a protected environment and supporting execution of encrypted code. The system has strong security guarantees. In particular, the cryptographic keys are never stored on the target machine, but rather delivered to it from a remote server, upon a successful verification of its authenticity. The keys and the decrypted instructions are protected by a thin hy- pervisor at all times. The system allows the encryption and execution of both native and Java code. During native code execution, the decrypted instructions are inaccessible to a potentially malicious code. This is achieved by either preventing execution of any other code or by protecting the memory region containing the decrypted instructions during their execution. Java programs, unlike native programs, are not executed directly by the processor, but are interpreted (and sometimes compiled) by the Java Virtual Machine (JVM). Therefore, the JVM will require the cryptographic key to decrypt the encrypted portions of Java code, and there is no feasible way of securing the key inside the JVM. The thesis proposes to implement a Java bytecode interpreter inside the secure environment, governed by a thin hypervisor. This interpreter will run in parallel to the standard JVM, both cooperating to execute encrypted Java programs.