Problem solve Get help with specific problems with your technologies, process and projects.

Fortifying Linux against common malware

Find out how to fight malware by teaming a new capability in the Linux kernel with processor-based tools.

Securing enterprise Linux desktops against hostile code has gotten easier, thanks to Ingo Molnar's work on the NX enabler patch in Linux kernel 2.6.8 and processor-based, page protection mechanisms. The trick is in the execution, and, as usual, the Microsoft way probably is not the right choice.

Common IT wisdom calls for measuring computational performance by execution cycles and runtime efficiency, so it seems counterintuitive to gauge a processor by what it does not execute. This approach, however, brings a much-needed preventive capability to the desktop arena, one that has been used successfully on servers to fight malware.

Although not entirely a new concept, both AMD's No-Execute (NX) and Intel's Execute Disabled (XD) capabilities provide an underlying page protection mechanism entrenched within the inner workings of microprocessor Page Table Entries (PTE). These page tables are essentially windows to information in memory regions that core operating system (OS) components use.

In the 64th bit of the PTE of the x86 architecture, there resides a Boolean flag that indicates whether or not a page is designated as executable, or as inactive, non-executable data. This concept has been practiced on the enterprise level for decades now in server processors such as Intel Itanium, Sun SPARC and even IBM PowerPC.

There are two primary camps into which execution-disabled page protection attributes fall; namely, hardware and software based solutions. Software solutions are primarily third-party additions to the host operating system itself and are not the focus of this article. Hardware solutions -- as provided by recent offerings by vendors such as AMD, Intel, Transmeta, and VIA -- are more robust and reliable and can be implemented in Linux without involving third-party software emulation.

Generically defined a no-execute page protection attribute segregates memory regions into active data (in the form of executable instruction streams) and inactive data (information parsed by active data). Any segment with the no-execution bit flagged is designated inactive, and therefore will not be interpreted by the processor as code.

Modern Intel-based (x86) processor designs make few distinctions between active code bits and inert data; only in the context of runtime execution does a processor know to handle information one way or the other. That said, such processors may be easily tricked into digesting handcrafted executable code when a faulty application permits data to be misrepresented and will gladly interpret a potentially hostile byte sequence as instructions.

While marketing machines for the Windows segment are pushing the non-executable bit as a preventive measure against a variety of malware forms (Trojans, worms, and viruses), the Unix/Linux realm should be concerned primarily about buffer overflows.

Unbounded string operations that store arguments and variables on the stack are especially suspect, since a simple overflow beyond the bounds of a stack-based array can easily trick the processor into interpreting unwanted code. This is where the non-execution flag comes into its own.

Non-executable stack and heap segments prevent object code from spilling over into areas meant to serve only as application data and, therefore, stop vanilla buffer overflow attempts dead. Exploit attempts that require stack or heap segments to be both writable and executable are rendered ineffective. As mentioned earlier, thanks to the NX enabler patch (available in Linux kernel version 2.6.8 and higher), you can take immediate advantage of processors that support this additional protection scheme.

When incorporated within a larger overall security framework, such as a series of compiler extensions and safe library call replacements, the non-executable protection mechanism enables a robust architecture that is more resilient to attacks than a typical M1 Abrams battle tank.

About the authors: Ed Tittel is a full-time freelance writer and trainer based in Austin, TX, who specializes in markup languages, information security, and IT certifications. Justin Korelc is a long-time Linux hacker who works with Ed, and concentrates on hardware and software security topics. Together, both contributed to a recent book on Home Theater PCs. Right now, both are busy at work on the Linux-based MythTV multimedia environment.

Dig Deeper on Linux servers

Start the conversation

Send me notifications when other members comment.

Please create a username to comment.