Software obfuscation is a
protection technique for making code unintelligible to automated program
comprehension and analysis tools. It works by performing semantic preserving transformations
such that the difficulty of automatically extracting the computational logic
out of code is increased. Obfuscating transforms in existing literature have been
designed with the ambitious goal of being resilient against all possible
reverse engineering attacks. Even though some of the constructions are based on
intractable computational problems, we do not know, in practice, how to
generate hard instances of obfuscated problems such that all forms of program
analyses would fail.
In this thesis, we address
the problem of software protection by developing a weaker notion of obfuscation
under which it is not required to guarantee an absolute blackbox
security. Using this notion, we develop provably-correct obfuscating transforms
using dependencies existing within program structures and indeterminacies in
communication characteristics between programs in a distributed computing
environment. We show how several well known static analysis tools can be used
for reverse engineering obfuscating transforms that derive resilience from
computationally hard problems. In particular, we restrict ourselves to one
common and potent static analysis tool, the static slicer, and use it as our
attack tool. We show the use of derived software engineering metrics to
indicate the degree of success or failure of a slicer attack on a piece of
obfuscated code. We address the issue of proving correctness of obfuscating
transforms by adapting existing proof techniques for functional program
refinement and communicating sequential processes.
The results of this thesis
could be used for future work in two ways: first, future researchers may extend
our proposed techniques to design obfuscations using a wider range of
dependencies that exist between dynamic program structures. Our restricted attack
model using one static analysis tool can also be relaxed and obfuscations
capable of withstanding a broader class of static and dynamic analysis attacks
could be developed based on the same principles. Secondly, our obfuscatory strength evaluation techniques could guide
anti-malware researchers in the development of tools to detect obfuscated strains
of polymorphic viruses.