Buffer overflow vulnerabilities exist since more than two decades and they are still frequently exploited by attackers. By exploiting these existing vulnerabilities, attackers are able to completely compromise target systems. In most cases because of only one unsecure line of code. However, the aspect that will be discussed in this thesis is related to a particular type of buffer overflows called stack-/heap-smashing. In detail, the behavior on the basis of modern operating system will be investigated. First of all the analysis will concentrate on basical theoretical aspects behind the concept of stack-/heap-smashing. The theoretical part includes considerations concerning the science of computer engineering like the functionality of CPU register, the construction of the stack layout and the process mechanism of the memory management. The purpose is to help the reader to understand the content of this document and make the subject less mysterious.
In a further step some of the protections that have been developed in order to counter and defeat stack-/heap-smashing will be considered. These countermeasures are provided at different levels ranging from hardware- to compiler-based.
The goal is to create a practical scenario in order to analyze the situation on Windows and Linux. In addition, some tricks will be tried and discussed to bypass these protections. Attempts will be carried out to exploit the vulnerabilities even if a countermeasure has been applied.