Randy Rayess
Reverse engineering can be defined as a process of understanding inner structure of a particular hardware or software based on an information, extracted from it. Software reverse engineering typically aims to answer the question of how does certain applications or systems work. Of course, when detailed documentation is available, the process becomes fairly easy. However, in practice, most of the time such information is very restricted, forcing you to use another way to understand the software in question.
There are many situations when software engineering is necessary:
- Researching and dissecting malware in order to learn how to detect it and best protect from it
- Researching network protocols to provide better network security
- Researching particular file format for storing particular data, such as disk images and mail bases
- Improving compatibility with a closed platforms or third-party software
- Researching undocumented platform features
- Testing protection of your own software from reverse engineering
This article covers the basics of how to reverse engineering software, touching on what knowledge is required and what tools you should use. Reverse engineering process requires extensive programming experience, as well as knowledge of a number of tools, necessary to take applications apart.
Knowledge required for software engineering
Reverse engineering different software often requires a widely different knowledge bases. For example, if you need to reverse engineer any network applications, you will need to understand the network structure, know about inter-process communication and understand the structure of network packets that you need to analyze, not to mention connections and their order.
Reverse engineering crypto algorithms requires deep understanding of crypto science, as well as extensive knowledge of the most popular encryption methods and algorithms they employ.
Reverse engineering file structure not only require knowledge on the file format, but also on how various system components interact with file in question. Such a task will be much easier for a person with extensive experience of working with files and file systems, as they will be able to incorporate this experience into the approach they take to reversing. One example of something, that experienced person will be able to do, is to create a test writing to the unique type values of the file, while at the same time logging the size of written data and offsets to the storage file. This may help to quickly find common patterns when it comes to calculating offset, which in turn will provide the necessary tip on how files internally structured.
While the specialized knowledge is very important, there is also a common information that anyone wondering how to reverse engineer should possess, such as various programming languages and compilers, as well as typical application structures.
Bread and butter of any software reverse engineering is disassembler. It allows to determine algorithms and logic that the software in question employs. However, with a myriad of compilers and file formats out there, not to mention widely different operating systems, there is no single tool that fits all situations, and you will need to find different approaches to different problems.
In order to better understand the decompiled code you will also need to know assembler, the way functions are called, understand the stack structure as well as know about the concept of stack frames, etc. Modern compilers excel at heavily optimizing the code, thus when you run it through a compiler and then decompile it, you will see a stark difference with original code. Thus, when reverse code engineering, it is necessary to not try to guess the original code, but rather to understand the idea behind it, why a certain part was written and what it is supposed to do. By knowing how assembler outputs various code, you will be able to think up some ideas. Next step is to just create your own prototype based on your understanding of the purpose of original code.
Tools for software reverse engineering
Disassemblers. Disassembler – is a software, designed to translate executables to assembly code. IDA Pro is the most famous disassembler that is very easy to recommend. It provides a number of tools that allow to greatly speedup reverse engineering, such as displaying the function call tree, parsing export and import of the executable and showing the necessary information about it, etc. It can even display code in C, which proves extremely helpful for anyone who is not intimately familiar with assembly.
Sysinternals utils. Windows software reverse engineering makes heavy use of all sysinternal tools:
- TCPView – this tool serves as a network sniffer, allowing to view data on TCP/UDP packets
- TDIMon – tool, that is similar to network sniffer, but conducts monitoring of a socket level
- PortMon – tool for monitoring physical serial and parallel ports, as well as any traffic that goes via them.
- WinObj – tool, that allows to display all objects inside the system in a neat hierarchical structure. It can prove extremely useful if you’re wondering how to reverse engineer driver, as well as software, that employs synchronization primitives
API Monitor. This tool allows to determine any called APIs and their behavior. Tool has a very powerful built-in database, containing the calls to a very large number of APIs, not only kernel32 and ntdll ones, but also managed environment, COM, and others.
Debuggers. Debugger is the necessary tool for any developer, allowing them to see how their own software behaves at any particular point. You can use debuggers in the same way for reverse engineering. The most popular ones out there are WinDBG and OllyDBG, the latter of which provides built-in disassembler and API parsing, while the former feature a kernel level debugging. Both very powerful tools, and choosing between them will entirely depend on your task at hand.
BIO
Dennis Turpitka, CEO of the Apriorit, is an expert within Digital Security solution business design and development, Virtualization and Cloud Computing R&D projects, establishment and management of Software Research direction. Successful entrepreneur, who organized several security start-ups.
