Buffer Overflow Exploitation In a simple program where the user is required to input their email address, we allocate a fixed amount of memory, let's say 100 bytes. However, we don't include any conditions to check if the user exceeds this size limit when entering their email address. The consequence of this oversight is that if the user enters an email address that's, let's say, 150 bytes long, the extra 50 bytes will overflow into memory allocated for other variables. This will disrupt the program's behavior, potentially causing unintended results. Essentially, this situation leads to a buffer overflow, which is a common security vulnerability and could be exploited by individuals with malicious intent.
Functions like printf and sprintf can be vulnerable if they are printing user input that isn't properly validated. This can allow attackers to read or write data from the stack, essentially leading to a stack overflow attack. Insecure library functions like gets and strcpy do not typically check the size of the buffer and are known to cause buffer overflow attacks.
An attacker could potentially manipulate the program by overwriting the return pointer to redirect it to a specific address of their choosing. This action is done to achieve their own objectives or gain an advantage.
In the traditional x86 architecture, the stack top is located at the lower memory address. Essentially the stack grows from the higher memory address to the lower address. So the stack tack begins at the higher memory address and grows downward to lower memory as more items are pushed into the stack as shown in the above diagram.
Stack based overflow vs heap-based overflow. When it comes to a stack-based overflow, the attacker targets the vulnerabilities in the stack memory, which stores functions and local variables. On the other hand, in a heap-based overflow, they target the heap memory, which is used for dynamic memory allocation through functions like malloc. It is usually harder for attackers to perform heap overflow attacks since the memory layout in the heap is very complex compared to that of the stack. As it contains objects that are dynamically allocated, it makes it hard for the attacker to follow and predict the addresses. In stack overflow attacks, the return address is typically overwritten to redirect program execution, whereas in heap overflow attacks, data structures and function pointers must be manipulated to control the program's behavior.
Per-system call stack randomization. As we know, Address Space Layout Randomization (ASLR) randomizes sections of the program's memory each time the program is executed, leading to changing addresses of functions, variables, and buffers with every run. Similarly, the per-system call stack randomization is built on a similar concept, where the randomization occurs each time a system call is made, rather than randomizing at the program startup like ASLR. So, every time a system call is made, the kernel's stack is randomized, leading to changing addresses of functions, variables, and buffers with every system call.
Whenever a system call is made, a transition is made from the user mode to the kernel mode in order to call the glibc library. So, in this process of mode switching, the user's context is saved before switching to the kernel. This is done so that after the system call, the saved registers can be stored to restore program execution smoothly. If the registers aren't saved before switching the mode from user to kernel, randomization of the stack might have an unstable impact on the program's execution, leading to a program crash.
malloc() function vulnerability
The malloc() function is a predefined function that essentially allocates memory above the heap since the heap uses dynamic memory allocation. malloc() stores data during runtime and returns the variable's address. So, the thing to keep in mind when using malloc() is that since it is used to store memory on the heap, that memory should be freed when no longer needed using a function called free(). So, if the user inputs data where that buffer is stored using malloc(), and the input data is larger than the buffer size, it can lead to buffer overflows and result in code execution. A dangling pointer is another vulnerability that can result from improper use of malloc(). In this scenario, if a pointer keeps trying to call or reference memory that has already been deallocated using the free() function, it can lead to unwanted program crashes or misuse by attackers for malicious purposes.