Buffer Overflow Protection

Several years of innovation in preventing against buffer overflow did not yielded effective results in preventing buffer overflow; solution against this problem cannot happen at one level and need to tackle at multi-level.

Developers are not aware of the existing methods to prevent stack based attacks, due to lack of awareness in developer level, these problems are still continuing.

Hardware based protection – Data Execution Prevention (DEP)

Intel and AMC has implemented feature which disables execution in Stack and Data segments. Intel markets this feature as XD (eXecute Disable) and AMD markets as NX (No eXecute), both has same functionality, only name is changed. AMD64 and Intel’s P4 codenamed Prescott (EMT64) supports this feature.

Identifying CPU supports this protection can be done using CPUID instruction. This protection is implemented using the existing reserve field of Page Table Entry (PTE), PTE is size of 64 bites and XD uses the last bit i.e., 63rd bit. If PAE (Physical address extension) is enabled then 63 bit is used as XD bit or else it will treat 63rd bit as reserved. If this bit is enabled for any page, then CPU cannot execute any code from that page.

It NX bit is enabled and if application is trying to execute code in stack or data segment where this bit is enabled, CPU will throw exception and leave to OS to handle it further.

Operating System based protection

Windows XP SP2 DEP based protection: This is OS implementation to prevent any execution of code in memory pages, heap, stack and other data sections. OS will prevent all kinds of attack which is based on Exception handling mechanism in windows. This will be supported in all machines if it runs XP and SP2. By default software based DEP will only protect limited system binaries.

C Runtime libraries

Generally, most of the C function which is provided to process buffer is vulnerable to attack. Many of these functions do not take into account secure coding guidelines. There is no internal checking’s for buffer overflow and all these function blindly does what you ask for without checking underflow or overflow.

Some of the C functions which is vulnerable are strcpy(), strcat(), sprintf() etc… and similarly you can find huge list, almost all of string functions in C/C++ library. Even the integer manipulation has the same problem. Unless programmer manually handles and there is no built-in way.

Visual studio 2005 removes most of these functions and make sure programmer passed the size of the buffer and even other compilers has implemented library which is safer. For ex: instead of strcpy(), it enforces to use strncpy().

Visual C/C++ – /GS option

GS compiler switch is a cookie which is placed in between the buffer and return address. After placing the cookie the stack will have following structure:

Function parameters

Function return address

Frame pointer

Cookie ( New data in stack of 4 bytes )

Exception handler frame

Locally declared variable

Callee saved registers

Cookie value: Cookie is a random value which is generated using the combination of the time, process id, thread id, tick count and performance counter. All these fields are XOR’d and used as cookie value and this is only 4 bytes value. This cookie should not be Zero and due the nature of generation it will be a random in different application. This generation happens when application begins and using C function __security_init_cookie().

After generating this 4 byte cookie value, it will assign to global cookie variable. In each function it will insert this value between the Buffer and return address. After executing the function it will check the local stored cookie value with global cookie value, if it’s not matching that means stack corruption has happened and application will halt by displaying following error message box:

C Example of Buffer overflow:

Code generated by compiler for C code is as follow:


Take a look at following disassembled code. Before executing the strcpy() function, stack was proper (read Marked rectangle) :


0x0012F9F8 is an ESP value and pointing to top of the stack. Stack value is as follow:

  • 0x00401e1d : Return address
  • 0x0012fa18 : EBP register value
  • 0xb2703066 : Cookie value
  • 0x008f07c4 & 0x00000212: are allocated to Buffer variable with extra alignment bytes. This value is a junk value since in our sample we just declared without assigning any value to it.
  • 0x00403c20: This is a address for a string “This is Testing” and this is stored in read only data segment.
  • 0x0012fa00 is an address of a Buffer variable which is passed to strcpy() function

After executing the instruction “CALL 004025F8” ( strcpy() function ), stack will look like following:


If you observe above figure and previous, you can easily identify the cookie value 0xb2703066 is overwritten. After calling function to compare global and local cookie, if there is any change it will call Buffer overflow message box.

Visual C/C++ – RunTime Checks (/RTC)

Runtime checking switch is a good addition to a latest version of VC++. This option works only for in non-optimized binary. But it works both on Debug and Release binaries (Although there is less chance that anyone is going to compiler the release binary without optimization). There are 3 switches /RTCs , /RTCu & /RTCc

Runtime check following error conditions:

RTCs : This switch helps in initializing all the local variables with 0xCC ( or INT 3 ). This helps to find certain run-time problems which happens only in release binary and not debug binary. It helps in stack pointer verification around every function call. This helps mainly in preventing the calling convention problem and it also helps checking overflow and underflow of all variables. This is done by adding 0xCCCCCCCC ( 4 bytes ) before and after the variable.


Error box clearly mentions which Buffer is corrupted.

RTCu : This switch helps in identifying the variables used without doing any initialization.

We are trying to copy pSrcPtr (un-initialized pointer) to Buffer variable. After executing the strcpy() function at the end of the function it throws above error.

RTCc : This switch helps in identifying if there is any data truncation. If you are assigning integer variable to char variable, you will get popup box of error.

We are trying to assign 0x800000 integer value to char type ‘c’. Once it reaches end of the function it will display above error box.

Deep inside RTCs to check how it is implemented by taking following code as sample:


Once it enters the function, it will make all the variables 0xCC including alignment space. Each and every variable is placed between 0xCC ( INT 3) byte. To achieve this functionality take a look at following disassembled code of RTCFunc() what kind of code compiler is going to add it:


If you have a variable and its value is 0x12345678. This is guarded in stack like:

0xCCCCCCCCCCCCCCCC (8bytes)
0x12345678(4bytes)
0xCCCCCCCCCCCCCCCC(8bytes)

After executing the function if any of the 0xCC byte is overwritten, then it is will identify which variable is corrupted and reports in error dialog Box. Take a look at following figure and when code is proper without overflow or underflow:


Blocked rectangle is a stack data. After executing the function to copy strings to all 3 buffers, you can see the data in stack in following order:

  • 0x004155C8 : Return address
  • 0x0012f308 : EBP register data
  • 0xCCCCCCCC : Bytes which is overwritten by compiler inserted code
  • 0x00434241 : BufferOver data of “ABC”
  • 0xCCCCCCCCCCCCCCCC: Bytes which is overwritten by compiler inserted code
  • 0x00454443 : BufferOver1 data of “CDE”
  • 0xCCCCCCCCCCCCCCCC: Bytes which is overwritten by compiler inserted code
  • 0x00484746 : BufferOver2 data of “FGH”
  • 0xCCCCCCCCCCCCCCCC: Bytes which is overwritten by compiler inserted code

Take a look at following code if there is any overflow or underflow how it overwrites. In this example I am going to 7 bytes of data to BufferOver variable:

Stack representation of overwritten data of above figure is as follow:

  • 0x004155C8 : Return address
  • 0x0012f308 : EBP register data
  • 0x004e4d4c is over written data instead of 0xCCCCCCCC. By checking this value with 0xCC compiler will identify which variable has corrupted
  • 0x00434241 : BufferOver data of “ABCKLMN” ( This variable is using the next bytes which is not reserved for this purpose.
  • 0xCCCCCCCCCCCCCCCC: Bytes which is overwritten by compiler inserted code
  • 0x00454443 : BufferOver1 data of “CDE”
  • 0xCCCCCCCCCCCCCCCC: Bytes which is overwritten by compiler inserted code
  • 0x00484746 : BufferOver2 data of “FGH”
  • 0xCCCCCCCCCCCCCCCC: Bytes which is overwritten by compiler inserted code

Following is the disassembled code for checking the variables for consistency of 0xCC bytes. Compiler has its own function of _RTC_CheckStackVars() function :

If you use all of these protection methods, it will only mitigate the risk of buffer overflow. But these are not fool proof methods. You might even need to use 3rd party Protection software’s. Most of these problems can be mitigated if the developer has the awareness about security risks during the development of the product.