In this article, I'll describe how to manage cluster with pacemaker, including creating cluster, managing the cluster, displaying cluster status, adding a node into cluster, and removing a node from the cluster.

Cluster creation

Start the pcsd daemon on each nodthe node

Run the following commands on each of the node, the first is to start pcsd daemon, the second is to make sure pcsd start after boot

#service pcsd start
#chkconfig pcsd on
also make sure cman and corosync are off on boot
#chkconfig cman off
#chkconfig corosync off

Authenticate the nodes that will cnstitute the cluster

Pcsd is openssl based daemon written in ruby, manages pcs authentication between nodes, the authentication files are located in /var/lib/pcsd.

During the package installation, a user called hacluster is created on every node. It is recommended that the password for user hacluster be the same on each node.

To authenticate pcs to pcs daemon on the nodes in the cluster, run

pcs cluster auth [node] [...] [-u username] [-p password]

For example, to authenticates user hacluster on nodeA for both nodeB and nodeC in the cluster run command below, the command prompts for the password for user hacluster on the cluster nodes.

root@nodeA ~]# pcs cluster auth nodeB nodeC
Username: hacluster
Password:
nodeB: Authorized
nodeC: Authorized

Note: the authentication only need to be done once, and the authorization tokens are stored in the file ~/.pcs/tokens (or /var/lib/pcsd/tokens).

Configure and sync the cluster nodes

Once you get authentication between nodes, then you can start configuring and starting the cluster nodes

pcs cluster setup [--start] [--local] --name cluster_ name node1 [node2] [...]

# pcs cluster setup --name fs_cluster nodeA nodeB nodeC
nodeA: Updated cluster.conf...
nodeB: Updated cluster.conf...
nodeC: Updated cluster.conf...

If the  --start option is used, the command will also start the cluster services on the specified nodes.

If the --all option is used, the command will start the cluster services on all the nodes in the cluster.

If the  --local option is used, the command will only modify the change on local node.

If no nodes specified, the cluster service will only start on local node(of course).

Managing the cluster

Start, stop cluster services on the cluster nodes

pcs cluster start|stop [--all] [node] [...]

The following commands start, stop, and force to stop cluster, if no node or --all specified, only local node cluster service will be affected.

To start cluster

#pcs cluster start --all
nodeA: Starting Cluster...
nodeB: Starting Cluster...
nodeC: Starting Cluster...

To show cluster status

# pcs cluster status
Cluster Status:
 Last updated: Mon Aug 31 21:40:52 2015
 Last change: Mon Aug 31 21:40:32 2015
 Stack: cman
 Current DC: nodeA - partition with quorum
 Version: 1.1.11-97629de
 3 Nodes configured
 0 Resources configured

PCSD Status:
  nodeA: Online
  nodeB: Online
  nodeC: Online

To stop cluster

#pcs cluster stop -all

To forcebly stop cluster service on local node, run

#pcs cluster kill

Enable and Disable cluster service

To config cluster to run on system starts on local node or all nodes,

# pcs cluster enable --all
nodeA: Cluster Enabled
nodeB: Cluster Enabled
nodeC: Cluster Enabled

To config cluster to stop on system starts on local node or all nodes

pcs cluster disable [--all] [node] [...]

Standby and Unstandby a node

To put a node to standby mode, so that it doesn't take any resource

pcs cluster standby [--all] [node] [...]

To put a node to unstandby mode, so it can takes back its resource

pcs cluster unstandby [--all] [node] [...]

Remove/destroy cluster configuration

To destroy/remove a cluster configuration, run the following commands

# pcs cluster destroy --all
nodeA: Successfully destroyed cluster
nodeB: Successfully destroyed cluster
nodeC: Successfully destroyed cluster
# pcs cluster status
Error: cluster is not currently running on this node

Displaying cluster status

Display  cluster status and resource

#pcs status

Display cluster status

#pcs cluster status

Display cluster resource

#pcs cluster resource

Add a node into the cluster

Install pacemaker cluster packages

# yum install pcs fence-agents pacemaker cman psmisc policycoreutils-python

To prevent corosync, cman start on boot

#chkconfig corosync off
#chkconfig cman off
#chkconfig

Set password for user hacluster and start pcsd

#passwd hacluster
#service pcsd start
#chkconfig pcsd on

Authenticate and add node on one of existing node in the cluster

#pcs cluster auth nodeD
# pcs cluster node add nodeD
nodeA: Corosync updated
nodeB: Corosync updated
nodeC: Corosync updated
nodeD: Updated cluster.conf...

Authenticate and add node on new node

#pcs cluster auth
#pcs cluster start
Starting Cluster...
#pcs cluste enable


# pcs cluster status
Cluster Status:
 Last updated: Mon Aug 31 22:09:05 2015
 Last change: Mon Aug 31 22:08:42 2015
 Stack: cman
 Current DC: nodeA - partition with quorum
 Version: 1.1.11-97629de
 4 Nodes configured
 0 Resources configured

PCSD Status:
  nodeA: Online
  nodeB: Online
  nodeC: Online
  nodeD: Online

Remove a node from the cluster

#remove resources that has to be on the node
#remove stonith agent to the node
#pcs cluster node remove nodeD
# pcs cluster status
Cluster Status:
 Last updated: Mon Aug 31 22:10:53 2015
 Last change: Mon Aug 31 22:10:32 2015
 Stack: cman
 Current DC: nodeA - partition with quorum
 Version: 1.1.11-97629de
 3 Nodes configured
 0 Resources configured

PCSD Status:
  nodeA: Online
  nodeB: Online
  nodeC: Online