So first, what's the difference? Solaris is an operating system that, with Trusted Extensions applied, runs the multi-level security (MLS) model. SELinux is a Mandatory Access Control scheme that can run on a number of Linux distributions and can include a policy that applies MLS.
What is MLS? MLS is a specific Mandatory Access Control scheme for particular types of tightly controlled environments such as the government and military. It is designed for environments with multi-layer data classifications and strict rules about how data is passed between those layers. You can read some more about MLS at this page
Solaris TX is aimed at implementing a trusted multi-level security (MLS) operating system to military and government standards - EAL4+/LSPP (see the common criteria portal for more information ). This creates a successor to the now defunct Trusted Solaris 8 variation of the Solaris operating system.
As described, SELinux is a Mandatory Access Control tool that is included on a number of Linux operating systems. On some of these operating systems - most notably Red Hat Enterprise Linux 5 - a security policy that implements MLS is also included. Enabling this policy on Red Hat Enterprise Linux 5 aims to allow the platform to be certified to the EAL4+/LSPP standard. But you can also load, extend or enhance other policy on a host in addition to the MLS policy. SELinux is designed to be extensible and potentially run multiple policies simultaneously.
Where does this leave us with for choice of an operating system? The decision becomes a little more complicated and actually comes down to:1. Solaris running Trusted Extensions, or 2. Linux distribution running SELinux
Without a view of your requirements and skills, it's hard to recommend one or the other. Assuming you have the know-how to run either Solaris or Linux, the purpose of the host is the focal concern. If you are dealing with a government client with a requirement for MLS then Solaris 10 TX may be the right approach. This is not to say Solaris TX isn't a good choice in other circumstances but it is designed for a specific purpose, and it is potentially best deployed for that purpose.
Alternatively you can choose a Linux-based distribution and run SELinux on it. SELinux is designed to have a much more flexible policy approach and can be more easily customized to manage other security controls. This maintenance may take more effort and overhead than Solaris TX but does allow you to deviate more readily from the MLS baseline.
Hope that helps clarify the situation. Here are some links to further discussion on SELinux versus Solaris TX:http://james-morris.livejournal.com/19377.html http://blogs.sun.com/gfaden/entry/comparing_selinux_with_solaris_trusted http://mentalrootkit.org/?p=16 James Turnbull is a Council member for Linux Australia - the peak advocacy body for Linux in Australia.
Dig Deeper on Linux servers
Related Q&A from James Turnbull
A user wants to implement OSSEC on a Windows server because he has no server side Linux operating system. Continue Reading
Configuring spam filters Spamassassin and dspam together in the email server Postfix is easy with the resources listed by our security expert. Continue Reading
If you had to choose between Bastille and SELinux, consider what you really need from a security program. A Linux expert explains the tradeoffs and ... Continue Reading