Manage Learn to apply best practices and optimize your operations.

Using Sysmask to safeguard Linux installations

Find out how Sysmask-protected Linux installations in the enterprise can reduce the number of exploitable kernel-level vulnerabilities.

Safeguarding against danger in an uncertain world is difficult. Cars, houses and businesses are all insured against a variety of natural and unnatural disasters, with insurance policies compensating victims of property damage after unavoidable mishaps.

On the enterprise IT level, insurance policies cannot cover unique server configurations, replace confidential data or compensate for a loss in productivity due to downtime. Therefore, to help make such assurances possible, organizations must use global policies that govern infrastructure operations in a comprehensive and proactive manner. For enterprises running Linux, a tool called sysmask may be the best preventive policy for exposed Linux assets this side of any DMZ.

A sysmask-protected Linux installation ensures a quantifiable level of protection against known (and unknown) vectors of attack as they occur. Sysmask creates a layer of separation between userland and kernel interaction based on a system-wide, event-driven system call policy. This reduces the number of exploitable conditions in the wild.

More security tips:

Fight spear phishing

Sealing Red Hat security gaps with open source security tools

Sysmask achieves granularity through easily defined policy files that are tailored according to the type of system call request (symlink, link, read, write, rename, etc.) and applied to running processes through the sysmask daemon (smkd). Since sysmask is implemented in the kernel, it gives comprehensive coverage to any application invoked through it and therefore requires neither recompilation of nor cooperation from a protected binary.

Sysmask consists of four integral components: an obligatory kernel patch and header file, a userland utility (/bin/sysmask), a daemon program initiated at system boot (/sbin/smkd), and configuration files to parameterize application behavior. A flexible configuration policy reacts to dynamic application calls, allowing system administrators to define a tightly controlled domain over workstation or server application behavior in much the same way that firewall rulesets restrict access to network objects. System calls effectively become triggers and activators when masked by the userland component, forcing the sysmask daemon to take corrective action against otherwise unchecked processes or unruly users.

While a complete SELinux (Security-Enhanced Linux) policy governs userland application behavior according to administrative policy, it falls short of protecting against kernel-level vulnerabilities. This leaves a small window of exposure for any exploitable vulnerability that resides beyond the coverage that SELinux policies provide. Sysmask takes a key position in the kernel code and implements a new structure type for each process. Just before any critical system call is made, that structure is checked against specified rules to make sure that applications operate within policy constraints.

The following code fragment comes from the sysmask patch against /usr/src/fs/exec.c for 2.6.12 series kernels:

@@ -1134,6 +1155,14 @@ int do_execve(char * filename,
  int retval;
  int i;
+ if(sm2_masked(SM2_EXECVE)) return -EPERM;
+ current->smk_got.ind = 0;
+ if(sm2_masked(SM2_EXEC)) {
+  retval = smk_checkexec(filename);
+  if(retval) return retval;
+ }
  retval = -ENOMEM;
  bprm = kmalloc(sizeof(*bprm), GFP_KERNEL);
  if (!bprm)

In the preceding kernel patch excerpt, it is evident that sysmask protection occurs before any critical system calls are invoked. In this code snippet, the do_execve call (which is wrapped in C by the execve function call) is checked against sysmask policy before the system proceeds with the real execve code portion.

While sysmask does not fully eliminate exploitable threats, for properly configured hosts, it does have two immediate benefits.

First, a sysmask-protected host is significantly more resilient to exploitation than a stock Linux kernel, raising the bar that a would-be attacker must scale in order to mount a successful exploit. With sysmask, those end-of-the-week advisories that often result in weekend-warrior exploitation will not be nearly as disruptive to system administrators on the following Monday morning.

Second, sysmask effectively extends Linux IT staff and administration response time between threat detection and removal while ensuring that vanilla exploitation schemes are rendered ineffective. Patches currently exist for Linux kernel versions 2.4.13 and 2.6.12, so you won't get to use sysmask with the latest stable revision (2.16.16) without heavy modification to both the patch and the utilities.

Ed Tittel is a full-time freelance writer and trainer based in Austin, Tex., who specializes in markup languages, information security and IT certifications. Justin Korelc is a long-time Linux hacker who works with Ed, concentrating on hardware and software security topics. Both contributed to a recent book on Home Theater PCs; right now, both are busy at work on a book about the Linux-based MythTV environment.

Dig Deeper on Linux servers

Start the conversation

Send me notifications when other members comment.

Please create a username to comment.