Safeguarding Linux HA server integrity with STONITH

Designing an approach to automatically shut down a failing node is an important step in protecting data and system integrity in a Linux HA cluster.

If multiple nodes in a high-availability cluster try to access a cluster file system at the same time, file system corruption can occur. As file system corruption may lead to data loss, it is something to be prevented at all costs. Therefore, before creating any services as resources in the high-availability (HA) cluster, an administrator should ensure cluster integrity using STONITH. This tip outlines the different options that an administrator should explore to ensure Linux HA server integrity.

The value of using STONITH
STONITH stands for “shoot the other node in the head.” It’s an extreme tactic that basically powers down the selected server remotely, removing it from the cluster and allowing other nodes in the cluster to take over. You would employ STONITH if one of the nodes in the cluster cannot be reached anymore by the other node(s) in the cluster. 

There are two reasons that could occur. Either the node is down and is not running the cluster resources it is responsible for, or the node isn't down but is no longer synchronized with other nodes in the cluster, still servicing its assigned cluster resources. This latter situation is known as the “split-brain scenario,” and may result in bad things happening to the cluster resources. Imagine, for example, a database that starts running twice in the cluster, or a file system that starts to be written between two independent nodes. Having a split brain in the cluster is bad, and the only way to ensure that no such scenario can occur in the cluster is by using the STONITH approach.

Hardware-based STONITH
In Linux HA Pacemaker clusters, STONITH is traditionally implemented by hardware solutions, such as a Dell Inc. Remote Access Card, a Hewlett-Packard Co. Integrated Lights-Out (iLO)management card that is installed in a server, or a manageable rack power distribution unit, where a Telnet command can be used to switch off power to a specific port. The secret behind the success of these solutions is that they all allow the cluster to talk to the physical server without involving  the operating system (OS), because the management solutions are on a different network by default.

In the case of an unresponsive node with an iLO management card, the iLO card has its own firmware OS and its own network connection, so even if the OS on the server is experiencing a crash, that doesn't affect the functioning of the server. Therefore, the cluster can use Telnet to send a command to the iLO board that switches off the server.

Disk-based STONITH
Although hardware-based STONITH works well, this approach requires specific hardware, which adds a degree of expense and vendor lock-in to the server nodes. This disadvantage doesn't exist for disk-based STONITH. The default disk-based STONITH in Linux HA Pacemaker clusters is known as split brain detection (SBD). To use this solution, you need a shared disk. As most clusters use a storage area network (SAN) to provide access to data, the SAN should be available on all nodes simultaneously, so the availability of shared disk devices normally isn't a problem. On this shared disk, the administrator creates a small partition that is used for SBD. In a normal situation, the cluster will be able to check availability of other nodes using the normal network connection. In case this default cluster communication fails, a poison pill is written to the SAN for the node that fails to reply. Since the Pacemaker software configuration requires a  failing node  to accept the poison pill, it will next terminate itself, thus ensuring a safe transition of resources in the cluster.

In this article, you've read why you should, in all cases, use STONITH to ensure the integrity of resources in the cluster. In Linux HA Pacemaker clusters, you can use either hardware- or disk-based STONITH solutions. A disk-based solution, such as SBD, is easy to implement, since this approach requires no specific hardware. In the next part of this article, you'll learn how to configure an SBD STONITH resource in a Linux HA Pacemaker cluster.

About the author: Sander van Vugt is an independent trainer and consultant living in the Netherlands. Van Vugt is an expert in Linux high availability, virtualization and performance and has completed several projects that implement all three. Sander is also a regular speaker on many Linux conferences all over the world. He is also the writer of various Linux-related books, such as Beginning the Linux Command LineBeginning Ubuntu Server Administration and Pro Ubuntu Server Administration.

Dig Deeper on Linux servers