Problem solve Get help with specific problems with your technologies, process and projects.

Securing Linux applications with compiler extensions

PaX team developers Justin Korelc and Ed Tittel explain how to secure your environment using compiler extensions as a first-line defense when running unverified 3rd-party software.

In-depth defense is a cardinal rule whenever implementing a secure IT framework. This is especially true of environments...

that operate largely dependent upon applications written using programming languages known not to be type-safe (such as C). A buggy application -- under the right conditions and with the correct permissions -- can be leveraged to an attacker's advantage and raises the specter of system-wide compromise.

Buffer overflow proofs of concept are produced with routine regularity, demonstrating how easy it can be to leverage unsafe coding practices into successful attacks and system takeovers. Thus, it's imperative to follow a layered methodology to fortify applications against the perils of unchecked code.

There are essentially three canons to follow to properly establish a secure framework when dealing with third-party software code bases, whose correctness and validity cannot always be known or proved conclusively:

  • Stack/Pointer Compiler Extensions (Stack-Smash Protector, StackGuard)
  • Run-time Library Checking (Libsafe)
  • Non-Executable Stack Regions (OpenBSD W^X, AMD NX bit)

This article focuses on using compiler extensions as a first-line of defense when running unverified third-party software in a secure environment. If utilized responsibly within a comprehensive security framework, damage resulting from erroneous coding constructs can be minimized. In later articles, the remaining layers of defense will be addressed, and should help to define an elementary approach to building a relatively secure infrastructure.

This first line of defense begins with recompilation of existing software code bases. This recompile uses specialized compiler extension functionality to safeguard against stack-based boundary violations. Although using these kinds of extensions is not a silver bullet, they can mitigate the effects of potentially harmful application side-effects arising from inexpert coding practices. Run-time library checking adds a secondary layer of defense, where a tertiary layer comes from the operating system and the microprocessor that ultimately executes the code (for example, the latest AMD and Intel 64-bit offerings both sport non-executable stack capability).

During program execution, the stack integrity instrumentation such as StackGuard or SSP calculates a random implementation-defined guard value, called a "canary," which is then inserted at the function prologue, just before the return address. A final integrity check of said canary value occurs just before function completion (at the function epilog). Any discrepancy between initial and final values causes immediate program termination.

A canary or canary word (x86 32-bit value) is a known-good sentinel value placed between a buffer and control data on the stack. Three types of canary values are widely used at present, including:

  • Terminator Canary : assumes that an overflow is based on unsafe string operations, which end with terminators (CR, LF, NULL, and EOF (-1)).
  • Random Canary: generated via some form of seed value inside a pseudo-random number generator, so that its value is non-deterministic (to ensure proper integrity).
  • Random XOR Canary : a combined value scrambled using an exclusive-or (XOR) operation on all or part of the control data; if either the canary or the control data gets clobbered, the canary value must then also change and will be wrong.

As a simple example, examine the following code snippet:

#include <stdint.h>
#include <stdlib.h>
#include <string.h>

extern char *__progname;
static volatile uint32_t guardv = 0xDEADC0DE;

int main (int argc, char **argv)
volatile uint32_t canary = guardv;
char buffer[128] = { 0 };

if (NULL == argv[1])
printf ("usage: %s \n", __progname);

/* unsafe string operation */
strcpy (buffer, argv[1]);

if (canary != guardv)
printf ("Corruption (aborting)\n");

printf ("buffer: %s\n", buffer);
return 0;

Owing to the stack layout as produced by the compiler on the architecture in this example (x86 32-bit Little Endian), the canary value occurs just after the buffer. Where the declaration for 'canary' occurs before 'buffer,' data consumed by 'buffer' will creep upward toward 'canary'. Overfilling this buffer thus invariably overwrites the canary value. In the demonstration that follows, a long user-supplied string successfully clobbers the canary value, which in turn halts the program.

[jkor@home workbench]$ /usr/bin/gcc –ggdb3 –ovuln vuln.c
[jkor@home workbench]$ ./vuln one2three4five6seven
Buffer: one2three4five6seven
[jkor@home workbench]$ ./vuln `perl -e 'print "X"x256'`
Corruption (aborting)

Though simple-minded, the code example above illustrates a way to verify stack frame control data after handling user-supplied input, before it can cause greater damage (should that input be crafted with malicious intent). Debugger output follows, providing a glimpse of the undesirable effects of unchecked input:

(gdb) set args `perl –e 'print "A"x256'`
(gdb) break main
(gdb) break strcpy
(gdb) run
... ...
# first breakpoint fires; single-step
(gdb) p/x canary
$1 = 0xdeadc0de
... ...
# second breakpoint fires
22 if (canary != guardv)
(gdb) p/x canary
(gdb) p/x guardv

Note that StackGuard does not protect against overwriting pointer and integer values also influencing program execution, which do not adversely affect the canary value (see PointGuard, part of StackGuard). Also, as the name implies, the stack-based nature of this protection scheme does nothing to mitigate heap-based exploitation.

For SSP, local variable reordering places buffer regions after pointer variables, and reinserts pointers in function arguments before local variable buffers. Both methods further refine the concept of stack protection instrumentation by proactively defending against further arbitrary memory corruption whenever such pointers might be overwritten.

The notion of checking one's work at runtime may sound deceptively simply but provides a powerful tool for detecting and escaping the potential ill effects of buffer overflow attacks.

About the authors: Ed Tittel is a full-time freelance writer and trainer based in Austin, Tex., who specializes in markup languages, information security, and IT certifications. Justin Korelc is a long-time Linux hacker who works with Ed and concentrates on hardware and software security topics. Together, both contributed to a recent book on Home Theater PCs and the Tom's Hardware 2005 Holiday Buyer's Guide.

This was last published in January 2006

Dig Deeper on Linux servers



Find more PRO+ content and other member only offers, here.

Start the conversation

Send me notifications when other members comment.

By submitting you agree to receive email from TechTarget and its partners. If you reside outside of the United States, you consent to having your personal data transferred to and processed in the United States. Privacy

Please create a username to comment.