Simple IP Failover Cluster with Centos 6 | it's all about documentation

for purpose of a more likely HA Icecast Cluster with two nodes I followed the steps below :
Prerequisites
1. Install Centos 6.5 with all the necessary applications – running on a KVM VM. Call this first installation node 1 (m1.dmd2.local)
2. Clone the first node and modify everything necessary to fire the 2nd VM up.

for the Installation of the Cluster Software I followed this instruction : Cluster Labs – The Home of Pacemaker.
All examples assume two nodes that are reachable by their short name and IP address:

node1 – 192.168.0.1
node2 – 192.168.0.2

The convention followed is that [ALL] # denotes a command that needs to be run on all cluster machines, and [ONE] # indicates a command that only needs to be run on one cluster host.
Next we use yum to install pacemaker and some other necessary packages we will need:

[ALL] # yum install pacemaker cman pcs ccs resource-agents

Configure Cluster Membership and Messaging

The supported stack on RHEL6 is based on CMAN, so thats what Pacemaker uses too.

We now create a CMAN cluster and populate it with some nodes. Note that the name cannot exceed 15 characters (we’ll use ‘pacemaker1’).

[ONE] # ccs -f /etc/cluster/cluster.conf --createcluster pacemaker1 
[ONE] # ccs -f /etc/cluster/cluster.conf --addnode node1 
[ONE] # ccs -f /etc/cluster/cluster.conf --addnode node2

Next we need to teach CMAN how to send it’s fencing requests to Pacemaker. We do this regardless of whether or not fencing is enabled within Pacemaker.

[ONE] # ccs -f /etc/cluster/cluster.conf --addfencedev pcmk agent=fence_pcmk [ONE] # ccs -f /etc/cluster/cluster.conf --addmethod pcmk-redirect node1 
[ONE] # ccs -f /etc/cluster/cluster.conf --addmethod pcmk-redirect node2 
[ONE] # ccs -f /etc/cluster/cluster.conf --addfenceinst pcmk node1 pcmk-redirect port=node1 
[ONE] # ccs -f /etc/cluster/cluster.conf --addfenceinst pcmk node2 pcmk-redirect port=node2

Now copy

/etc/cluster/cluster.conf

to all the other nodes that will be part of the cluster.

Start the Cluster
CMAN was originally written for rgmanager and assumes the cluster should not start until the node has quorum, so before we try to start the cluster, we need to disable this behavior:

[ALL] # echo "CMAN_QUORUM_TIMEOUT=0" >> /etc/sysconfig/cman

Now, on each machine, run:

[ALL] # service cman start 
[ALL] # service pacemaker start

A note for users of prior RHEL versions
The original cluster shell (crmsh) is no longer available on RHEL. To help people make the transition there is a quick reference guide for those wanting to know what the pcs equivalent is for various crmsh commands.

Set Cluster Options
With so many devices and possible topologies, it is nearly impossible to include Fencing in a document like this. For now we will disable it.

[ONE] # pcs property set stonith-enabled=false

One of the most common ways to deploy Pacemaker is in a 2-node configuration. However quorum as a concept makes no sense in this scenario (because you only have it when more than half the nodes are available), so we’ll disable it too.

[ONE] # pcs property set no-quorum-policy=ignore

For demonstration purposes, we will force the cluster to move services after a single failure:

[ONE] # pcs resource defaults migration-threshold=1

Add a Resource
Virtual IP address :
The next step is to actually let our cluster do something. We will add a virtual IP to our cluster. This virtual IP is the IP address that which will be contacted to reach the services (the icecast Server in our case). A virtual IP is a resource. To add the resource:

[node1]$ pcs resource create virtual_ip ocf:heartbeat:IPaddr2 ip=192.168.0.100 cidr_netmask=32 op monitor interval=1s
[node1]$ pcs status resources
virtual_ip (ocf::heartbeat:IPaddr2): Started

UPGRADE OS :
before upgrading to antother OS Level:
1. move resource to another node :

pcs resource move virtual_ip m2 #oder m1

2. put the passive node into standy mode

this is it :
pcs cluster standby m2
------------------------------------
# pcs resource standby virtual_ip m2

3. stop pacemaker and corosync service

service cman stop && service pacemaker stop

4. upgrade Node
5. reboot Node
6. put the upgraded passive node back into the quorum (unstandby)

pcs cluster unstandby m2

7. move the resource from the currently active node to the upgraded passive node

pcs resource move virtual_ip m1 #oder m2

8. for the second node follow the same steps 2. -6.

Other helpfull links :
quick reference guide for PCS
Building a high-available failover cluster with Pacemaker, Corosync & PCS
Active/Passive Cluster with Pacemaker, Corosync