[ULTRAMONKEY-USERS] Problem with Streamline High Availability and Load Balancing

Pierguido pierg75 at yahoo.it
Mon Oct 8 23:40:25 EST 2007


Hi all.
I'm trying to configure a 2 nodes cluster.
I'm trying to follow the instructions at this page:
http://www.ultramonkey.org/3/topologies/sl-ha-lb-eg.html
but cannot make the servers to get the VIP on the eth0 interface.
I tried everything but with no results.
I use debian etch.
Here are my configs:

/etc/ha.d/haresources:

srv-cluster-1 \
ldirectord::ldirectord.cf LVSSyncDaemonSwap::master \
IPaddr2::192.168.30.253/24/eth0/192.168.30.255 \
IPaddr2::192.168.30.252/24/eth0/192.168.30.255 \
drbddisk::mysql Filesystem::/dev/drbd0::/mnt/mysql::ext3::defaults \
drbddisk::www Filesystem::/dev/drbd2::/mnt/www::ext3::defaults \
mysql apache2

/etc/ha.d/ldiretord.cf:

# Global Directives
checktimeout=10
checkinterval=2
autoreload=no
logfile="local0"
quiescent=yes

# Virtual Server for HTTP
virtual=192.168.30.253:80
         fallback=127.0.0.1:80
         real=192.168.30.120:80 gate
         real=192.168.30.121:80 gate
         service=http
         request="index.html"
         receive="Test Page"
         scheduler=rr
         protocol=tcp
         checktype=negotiate

# Virtual Service for HTTPS
virtual=192.168.30.253:443
         fallback=127.0.0.1:443
         real=192.168.30.120:443 gate
         real=192.168.30.121:443 gate
         service=https
         request="index.html"
         receive="Test Page"
         scheduler=rr
         protocol=tcp
         checktype=negotiate

virtual=192.168.30.253:143
         fallback=127.0.0.1:143
         real=192.168.30.120:143 gate
         real=192.168.30.121:143 gate
         service=imap
         scheduler=wlc
         persistent=600
         protocol=tcp

virtual=192.168.30.253:993
         fallback=127.0.0.1:993
         real=192.168.30.120:993 gate
         real=192.168.30.121:993 gate
         service=imaps
         scheduler=wlc
         persistent=600
         protocol=tcp

# Virtual Service for POP
virtual=192.168.30.253:110
         fallback=127.0.0.1:110
         real=192.168.30.120:110 gate
         real=192.168.30.121:110 gate
         service=pop
         scheduler=wlc
         protocol=tcp
         persistent=600

# Virtual Service for SMTP
virtual=192.168.30.253:25
         fallback=127.0.0.1:25
         real=192.168.30.120:25 gate
         real=192.168.30.121:25 gate
         service=smtp
         persistent=600
         scheduler=wlc
         protocol=tcp


ipvsadm -L -n:

Prot LocalAddress:Port Scheduler Flags
   -> RemoteAddress:Port           Forward Weight ActiveConn InActConn
TCP  192.168.30.253:993 wlc persistent 600
   -> 192.168.30.120:993           Local   0      0          0
   -> 192.168.30.121:993           Route   0      0          0
   -> 127.0.0.1:993                Local   1      0          0
TCP  192.168.30.253:443 rr
   -> 192.168.30.121:443           Route   0      0          0
   -> 192.168.30.120:443           Local   0      0          0
   -> 127.0.0.1:443                Local   1      0          0
TCP  192.168.30.253:143 wlc persistent 600
   -> 192.168.30.121:143           Route   0      0          0
   -> 192.168.30.120:143           Local   1      0          0
TCP  192.168.30.253:110 wlc persistent 600
   -> 192.168.30.120:110           Local   0      0          0
   -> 192.168.30.121:110           Route   0      0          0
   -> 127.0.0.1:110                Local   1      0          0
TCP  192.168.30.253:80 rr
   -> 192.168.30.121:80            Route   0      0          0
   -> 192.168.30.120:80            Local   0      0          0
   -> 127.0.0.1:80                 Local   1      0          0
TCP  192.168.30.253:25 wlc persistent 600
   -> 192.168.30.120:25            Local   0      0          0
   -> 192.168.30.121:25            Route   0      0          0
   -> 127.0.0.1:25                 Local   1      0          0

ip addr sh:

1: lo: <LOOPBACK,UP,10000> mtu 16436 qdisc noqueue
     link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
     inet 127.0.0.1/8 scope host lo
     inet 192.168.30.253/32 scope global lo:0
2: eth2: <BROADCAST,MULTICAST,UP,10000> mtu 1500 qdisc pfifo_fast qlen 1000
     link/ether 00:1b:78:57:19:60 brd ff:ff:ff:ff:ff:ff
     inet 172.16.0.1/24 brd 172.16.0.255 scope global eth2
3: eth1: <BROADCAST,MULTICAST,UP,10000> mtu 1500 qdisc pfifo_fast qlen 1000
     link/ether 00:1b:78:57:19:61 brd ff:ff:ff:ff:ff:ff
     inet 10.0.0.1/24 brd 10.0.0.255 scope global eth1
4: eth0: <BROADCAST,MULTICAST,UP,10000> mtu 1500 qdisc pfifo_fast qlen 1000
     link/ether 00:18:fe:77:64:f8 brd ff:ff:ff:ff:ff:ff
     inet 192.168.30.120/24 brd 192.168.30.255 scope global eth0
     inet 192.168.30.252/24 brd 192.168.30.255 scope global secondary eth0



/var/log/syslog:

Oct  8 17:10:59 srv-cluster-1 heartbeat: [3294]: info: Initial resource 
acquisition complete (T_RESOURCES(them))
Oct  8 17:10:59 srv-cluster-1 ipfail: [3388]: debug: [We are srv-cluster-1]
Oct  8 17:10:59 srv-cluster-1 ipfail: [3388]: debug: auto_failback -> 0 
(off)
Oct  8 17:10:59 srv-cluster-1 ipfail: [3388]: debug: Setting message 
filter mode
Oct  8 17:10:59 srv-cluster-1 ipfail: [3388]: debug: Starting node walk
Oct  8 17:11:00 srv-cluster-1 ipfail: [3388]: debug: Cluster node: 
192.168.30.1: status: ping
Oct  8 17:11:00 srv-cluster-1 ipfail: [3388]: debug: Cluster node: 
srv-cluster-2: status: active
Oct  8 17:11:01 srv-cluster-1 ipfail: [3388]: debug: [They are 
srv-cluster-2]
Oct  8 17:11:01 srv-cluster-1 ipfail: [3388]: debug: Cluster node: 
srv-cluster-1: status: active
Oct  8 17:11:01 srv-cluster-1 ipfail: [3388]: debug: Setting message signal
Oct  8 17:11:01 srv-cluster-1 ipfail: [3388]: debug: Waiting for messages...
Oct  8 17:11:02 srv-cluster-1 ipfail: [3388]: debug: Other side is now 
stable.
Oct  8 17:11:02 srv-cluster-1 ipfail: [3388]: debug: Other side is now 
stable.
Oct  8 17:11:02 srv-cluster-1 ipfail: [3388]: debug: Got asked for num_ping.
Oct  8 17:11:03 srv-cluster-1 ipfail: [3388]: debug: Found ping node 
192.168.30.1!
Oct  8 17:11:03 srv-cluster-1 ipfail: [3388]: info: Ping node count is 
balanced.
Oct  8 17:11:03 srv-cluster-1 ipfail: [3388]: debug: Abort message sent.
Oct  8 17:11:24 srv-cluster-1 modprobe: WARNING: Not loading blacklisted 
module ipv6
Oct  8 17:11:42 srv-cluster-1 modprobe: WARNING: Not loading blacklisted 
module ipv6
Oct  8 17:12:28 srv-cluster-1 ipfail: [3388]: debug: Other side is unstable.
Oct  8 17:12:39 srv-cluster-1 kernel: drbd2: peer( Primary -> Secondary )
Oct  8 17:12:42 srv-cluster-1 kernel: drbd0: peer( Primary -> Secondary )
Oct  8 17:12:56 srv-cluster-1 heartbeat: [3294]: info: Received shutdown 
notice from 'srv-cluster-2'.
Oct  8 17:12:56 srv-cluster-1 heartbeat: [3294]: info: Resources being 
acquired from srv-cluster-2.
Oct  8 17:12:56 srv-cluster-1 heartbeat: [3294]: debug: 
StartNextRemoteRscReq(): child count 1
Oct  8 17:12:56 srv-cluster-1 heartbeat: [3447]: info: acquire all HA 
resources (standby).
Oct  8 17:12:56 srv-cluster-1 ResourceManager[3468]: [3478]: info: 
Acquiring resource group: srv-cluster-1 ldirectord::ldirectord.cf 
LVSSyncDaemonSwap::master IPaddr2::192.168.30.253/24/eth0/192.168.30.255 
IPaddr2::192.168.30.252/24/eth0/192.168.30.255 drbddisk::mysql 
Filesystem::/dev/drbd0::/mnt/mysql::ext3::defaults drbddisk::www 
Filesystem::/dev/drbd2::/mnt/www::ext3::defaults mysql apache2
Oct  8 17:12:56 srv-cluster-1 ldirectord[3493]: Invoking ldirectord 
invoked as: /etc/ha.d/resource.d/ldirectord ldirectord.cf status
Oct  8 17:12:56 srv-cluster-1 ldirectord[3493]: ldirectord is stopped 
for /etc/ha.d/ldirectord.cf
Oct  8 17:12:56 srv-cluster-1 ldirectord[3493]: Exiting with exit_status 
3: Exiting from ldirectord status
Oct  8 17:12:56 srv-cluster-1 ldirectord[3505]: Invoking ldirectord 
invoked as: /etc/ha.d/resource.d/ldirectord ldirectord.cf status
Oct  8 17:12:56 srv-cluster-1 ldirectord[3505]: ldirectord is stopped 
for /etc/ha.d/ldirectord.cf
Oct  8 17:12:56 srv-cluster-1 ldirectord[3505]: Exiting with exit_status 
3: Exiting from ldirectord status
Oct  8 17:12:56 srv-cluster-1 heartbeat: [3448]: info: Local Resource 
acquisition completed.
Oct  8 17:12:56 srv-cluster-1 ResourceManager[3468]: [3519]: info: 
Running /etc/ha.d/resource.d/ldirectord ldirectord.cf start
Oct  8 17:12:56 srv-cluster-1 heartbeat: [3294]: debug: 
StartNextRemoteRscReq(): child count 2
Oct  8 17:12:56 srv-cluster-1 heartbeat: [3294]: debug: 
StartNextRemoteRscReq(): child count 1
Oct  8 17:12:56 srv-cluster-1 ResourceManager[3468]: [3522]: debug: 
Starting /etc/ha.d/resource.d/ldirectord ldirectord.cf start
Oct  8 17:12:56 srv-cluster-1 ldirectord[3523]: Invoking ldirectord 
invoked as: /etc/ha.d/resource.d/ldirectord ldirectord.cf start
Oct  8 17:12:56 srv-cluster-1 ldirectord[3523]: Starting Linux Director 
v1.77.2.51 as daemon
Oct  8 17:12:56 srv-cluster-1 ResourceManager[3468]: [3527]: debug: 
/etc/ha.d/resource.d/ldirectord ldirectord.cf start done. RC=0
Oct  8 17:12:56 srv-cluster-1 ldirectord[3525]: Added virtual server: 
192.168.30.253:80
Oct  8 17:12:56 srv-cluster-1 ldirectord[3525]: Added virtual server: 
192.168.30.253:443
Oct  8 17:12:56 srv-cluster-1 ldirectord[3525]: Added virtual server: 
192.168.30.253:143
Oct  8 17:12:56 srv-cluster-1 ldirectord[3525]: Added virtual server: 
192.168.30.253:993
Oct  8 17:12:56 srv-cluster-1 ResourceManager[3468]: [3609]: info: 
Running /etc/ha.d/resource.d/LVSSyncDaemonSwap master start
Oct  8 17:12:56 srv-cluster-1 ResourceManager[3468]: [3611]: debug: 
Starting /etc/ha.d/resource.d/LVSSyncDaemonSwap master start
Oct  8 17:12:56 srv-cluster-1 ldirectord[3525]: Added virtual server: 
192.168.30.253:110
Oct  8 17:12:56 srv-cluster-1 ldirectord[3525]: Added virtual server: 
192.168.30.253:25
Oct  8 17:12:56 srv-cluster-1 ldirectord[3525]: Added fallback server: 
127.0.0.1:80 ( x 192.168.30.253:80) (Weight set to 1)
Oct  8 17:12:56 srv-cluster-1 kernel: IPVS: sync thread started: state = 
MASTER, mcast_ifn = eth0, syncid = 0
Oct  8 17:12:56 srv-cluster-1 ldirectord[3525]: Added fallback server: 
127.0.0.1:443 ( x 192.168.30.253:443) (Weight set to 1)
Oct  8 17:12:56 srv-cluster-1 LVSSyncDaemonSwap[3615]: [3684]: info: 
ipvs_syncmaster up
Oct  8 17:12:56 srv-cluster-1 LVSSyncDaemonSwap[3615]: [3686]: info: 
ipvs_syncmaster obtained
Oct  8 17:12:56 srv-cluster-1 ResourceManager[3468]: [3689]: debug: 
/etc/ha.d/resource.d/LVSSyncDaemonSwap master start done. RC=0
Oct  8 17:12:56 srv-cluster-1 ldirectord[3525]: Added fallback server: 
127.0.0.1:143 ( x 192.168.30.253:143) (Weight set to 1)
Oct  8 17:12:56 srv-cluster-1 ldirectord[3525]: Added fallback server: 
127.0.0.1:993 ( x 192.168.30.253:993) (Weight set to 1)
Oct  8 17:12:56 srv-cluster-1 ldirectord[3525]: Added fallback server: 
127.0.0.1:110 ( x 192.168.30.253:110) (Weight set to 1)
Oct  8 17:12:56 srv-cluster-1 ldirectord[3525]: Added fallback server: 
127.0.0.1:25 ( x 192.168.30.253:25) (Weight set to 1)
Oct  8 17:12:56 srv-cluster-1 ldirectord[3525]: Quiescent real server: 
192.168.30.120:443 ( x 192.168.30.253:443) (Weight set to 0)
Oct  8 17:12:56 srv-cluster-1 ldirectord[3525]: Quiescent real server: 
192.168.30.121:110 ( x 192.168.30.253:110) (Weight set to 0)
Oct  8 17:12:56 srv-cluster-1 ldirectord[3525]: Quiescent real server: 
192.168.30.121:993 ( x 192.168.30.253:993) (Weight set to 0)
Oct  8 17:12:56 srv-cluster-1 ldirectord[3525]: Quiescent real server: 
192.168.30.121:25 ( x 192.168.30.253:25) (Weight set to 0)
Oct  8 17:12:56 srv-cluster-1 ldirectord[3525]: Quiescent real server: 
192.168.30.120:143 ( x 192.168.30.253:143) (Weight set to 0)
Oct  8 17:12:56 srv-cluster-1 IPaddr2[3711]: [3935]: INFO: IPaddr2 
Resource is stopped
Oct  8 17:12:56 srv-cluster-1 ldirectord[3525]: Quiescent real server: 
192.168.30.121:143 ( x 192.168.30.253:143) (Weight set to 0)
Oct  8 17:12:56 srv-cluster-1 ResourceManager[3468]: [3959]: info: 
Running /etc/ha.d/resource.d/IPaddr2 
192.168.30.253/24/eth0/192.168.30.255 start
Oct  8 17:12:56 srv-cluster-1 ResourceManager[3468]: [3961]: debug: 
Starting /etc/ha.d/resource.d/IPaddr2 
192.168.30.253/24/eth0/192.168.30.255 start
Oct  8 17:12:56 srv-cluster-1 ldirectord[3525]: Quiescent real server: 
192.168.30.120:110 ( x 192.168.30.253:110) (Weight set to 0)
Oct  8 17:12:56 srv-cluster-1 ldirectord[3525]: Quiescent real server: 
192.168.30.120:80 ( x 192.168.30.253:80) (Weight set to 0)
Oct  8 17:12:56 srv-cluster-1 ldirectord[3525]: Quiescent real server: 
192.168.30.121:443 ( x 192.168.30.253:443) (Weight set to 0)
Oct  8 17:12:56 srv-cluster-1 ldirectord[3525]: Quiescent real server: 
192.168.30.120:993 ( x 192.168.30.253:993) (Weight set to 0)
Oct  8 17:12:56 srv-cluster-1 ldirectord[3525]: Quiescent real server: 
192.168.30.120:25 ( x 192.168.30.253:25) (Weight set to 0)
Oct  8 17:12:56 srv-cluster-1 ldirectord[3525]: Quiescent real server: 
192.168.30.121:80 ( x 192.168.30.253:80) (Weight set to 0)
Oct  8 17:12:56 srv-cluster-1 IPaddr2[4106]: [4153]: INFO: 
/usr/lib/heartbeat/send_arp -i 200 -r 5 -p 
/var/run/heartbeat/rsctmp/send_arp/send_arp-192.168.30.253 eth0 
192.168.30.253 auto 192.168.30.253 ffffffffffff
Oct  8 17:12:56 srv-cluster-1 IPaddr2[3965]: [4157]: INFO: IPaddr2 Success
Oct  8 17:12:56 srv-cluster-1 ResourceManager[3468]: [4158]: debug: 
/etc/ha.d/resource.d/IPaddr2 192.168.30.253/24/eth0/192.168.30.255 start 
done. RC=0
Oct  8 17:12:56 srv-cluster-1 IPaddr2[4170]: [4276]: INFO: IPaddr2 
Resource is stopped
Oct  8 17:12:56 srv-cluster-1 ResourceManager[3468]: [4290]: info: 
Running /etc/ha.d/resource.d/IPaddr2 
192.168.30.252/24/eth0/192.168.30.255 start
Oct  8 17:12:56 srv-cluster-1 ResourceManager[3468]: [4291]: debug: 
Starting /etc/ha.d/resource.d/IPaddr2 
192.168.30.252/24/eth0/192.168.30.255 start
Oct  8 17:12:57 srv-cluster-1 imapd: Connection, ip=[::ffff:192.168.30.120]
Oct  8 17:12:57 srv-cluster-1 imapd: LOGOUT, ip=[::ffff:192.168.30.120], 
rcvd=21, sent=354
Oct  8 17:12:57 srv-cluster-1 IPaddr2[4374]: [4401]: INFO: /sbin/ip -f 
inet addr add 192.168.30.252/24 brd 192.168.30.255 dev eth0
Oct  8 17:12:57 srv-cluster-1 IPaddr2[4374]: [4406]: INFO: /sbin/ip link 
set eth0 up
Oct  8 17:12:57 srv-cluster-1 IPaddr2[4374]: [4411]: INFO: 
/usr/lib/heartbeat/send_arp -i 200 -r 5 -p 
/var/run/heartbeat/rsctmp/send_arp/send_arp-192.168.30.252 eth0 
192.168.30.252 auto 192.168.30.252 ffffffffffff
Oct  8 17:12:57 srv-cluster-1 IPaddr2[4292]: [4416]: INFO: IPaddr2 Success
Oct  8 17:12:57 srv-cluster-1 ResourceManager[3468]: [4418]: debug: 
/etc/ha.d/resource.d/IPaddr2 192.168.30.252/24/eth0/192.168.30.255 start 
done. RC=0
Oct  8 17:12:57 srv-cluster-1 ldirectord[3525]: Restored real server: 
192.168.30.120:143 ( x 192.168.30.253:143) (Weight set to 1)
Oct  8 17:12:57 srv-cluster-1 ldirectord[3525]: Deleted fallback server: 
127.0.0.1:143 ( x 192.168.30.253:143)
Oct  8 17:12:57 srv-cluster-1 ResourceManager[3468]: [4465]: info: 
Running /etc/ha.d/resource.d/drbddisk mysql start
Oct  8 17:12:57 srv-cluster-1 ResourceManager[3468]: [4466]: debug: 
Starting /etc/ha.d/resource.d/drbddisk mysql start
Oct  8 17:12:57 srv-cluster-1 ldirectord[3525]: Restored real server: 
192.168.30.121:143 ( x 192.168.30.253:143) (Weight set to 1)
Oct  8 17:12:57 srv-cluster-1 kernel: drbd0: role( Secondary -> Primary )
Oct  8 17:12:57 srv-cluster-1 kernel: drbd0: Writing meta data super 
block now.
Oct  8 17:12:57 srv-cluster-1 ResourceManager[3468]: [4484]: debug: 
/etc/ha.d/resource.d/drbddisk mysql start done. RC=0
Oct  8 17:12:57 srv-cluster-1 Filesystem[4560]: [4566]: INFO: Running 
status for /dev/drbd0 on /mnt/mysql
Oct  8 17:12:57 srv-cluster-1 Filesystem[4560]: [4570]: INFO: /mnt/mysql 
is unmounted (stopped)
Oct  8 17:12:57 srv-cluster-1 Filesystem[4496]: [4571]: INFO: Filesystem 
Resource is stopped
Oct  8 17:12:57 srv-cluster-1 ResourceManager[3468]: [4585]: info: 
Running /etc/ha.d/resource.d/Filesystem /dev/drbd0 /mnt/mysql ext3 
defaults start
Oct  8 17:12:57 srv-cluster-1 ResourceManager[3468]: [4586]: debug: 
Starting /etc/ha.d/resource.d/Filesystem /dev/drbd0 /mnt/mysql ext3 
defaults start
Oct  8 17:12:57 srv-cluster-1 Filesystem[4651]: [4657]: INFO: Running 
start for /dev/drbd0 on /mnt/mysql
Oct  8 17:12:57 srv-cluster-1 kernel: kjournald starting.  Commit 
interval 5 seconds
Oct  8 17:12:57 srv-cluster-1 kernel: EXT3-fs warning: maximal mount 
count reached, running e2fsck is recommended
Oct  8 17:12:57 srv-cluster-1 kernel: EXT3 FS on drbd0, internal journal
Oct  8 17:12:57 srv-cluster-1 kernel: EXT3-fs: mounted filesystem with 
ordered data mode.
Oct  8 17:12:57 srv-cluster-1 Filesystem[4587]: [4667]: INFO: Filesystem 
Success
Oct  8 17:12:57 srv-cluster-1 ResourceManager[3468]: [4668]: debug: 
/etc/ha.d/resource.d/Filesystem /dev/drbd0 /mnt/mysql ext3 defaults 
start done. RC=0
Oct  8 17:12:57 srv-cluster-1 ResourceManager[3468]: [4696]: info: 
Running /etc/ha.d/resource.d/drbddisk www start
Oct  8 17:12:57 srv-cluster-1 ResourceManager[3468]: [4697]: debug: 
Starting /etc/ha.d/resource.d/drbddisk www start
Oct  8 17:12:57 srv-cluster-1 kernel: drbd2: role( Secondary -> Primary )
Oct  8 17:12:57 srv-cluster-1 kernel: drbd2: Writing meta data super 
block now.
Oct  8 17:12:57 srv-cluster-1 ResourceManager[3468]: [4701]: debug: 
/etc/ha.d/resource.d/drbddisk www start done. RC=0
Oct  8 17:12:57 srv-cluster-1 Filesystem[4777]: [4783]: INFO: Running 
status for /dev/drbd2 on /mnt/www
Oct  8 17:12:57 srv-cluster-1 Filesystem[4777]: [4787]: INFO: /mnt/www 
is unmounted (stopped)
Oct  8 17:12:57 srv-cluster-1 Filesystem[4713]: [4788]: INFO: Filesystem 
Resource is stopped
Oct  8 17:12:57 srv-cluster-1 ResourceManager[3468]: [4802]: info: 
Running /etc/ha.d/resource.d/Filesystem /dev/drbd2 /mnt/www ext3 
defaults start
Oct  8 17:12:57 srv-cluster-1 ResourceManager[3468]: [4803]: debug: 
Starting /etc/ha.d/resource.d/Filesystem /dev/drbd2 /mnt/www ext3 
defaults start
Oct  8 17:12:57 srv-cluster-1 Filesystem[4868]: [4874]: INFO: Running 
start for /dev/drbd2 on /mnt/www
Oct  8 17:12:57 srv-cluster-1 kernel: kjournald starting.  Commit 
interval 5 seconds
Oct  8 17:12:57 srv-cluster-1 kernel: EXT3-fs warning: maximal mount 
count reached, running e2fsck is recommended
Oct  8 17:12:57 srv-cluster-1 kernel: EXT3 FS on drbd2, internal journal
Oct  8 17:12:57 srv-cluster-1 kernel: EXT3-fs: mounted filesystem with 
ordered data mode.
Oct  8 17:12:57 srv-cluster-1 Filesystem[4804]: [4884]: INFO: Filesystem 
Success
Oct  8 17:12:57 srv-cluster-1 ResourceManager[3468]: [4885]: debug: 
/etc/ha.d/resource.d/Filesystem /dev/drbd2 /mnt/www ext3 defaults start 
done. RC=0
Oct  8 17:12:57 srv-cluster-1 ResourceManager[3468]: [4913]: info: 
Running /etc/init.d/mysql  start
Oct  8 17:12:57 srv-cluster-1 ResourceManager[3468]: [4914]: debug: 
Starting /etc/init.d/mysql  start
Oct  8 17:12:58 srv-cluster-1 mysqld_safe[4974]: started
Oct  8 17:12:58 srv-cluster-1 mysqld[4977]: 071008 17:12:58  InnoDB: 
Started; log sequence number 0 3727511
Oct  8 17:12:58 srv-cluster-1 mysqld[4977]: 071008 17:12:58 [Note] 
/usr/sbin/mysqld: ready for connections.
Oct  8 17:12:58 srv-cluster-1 mysqld[4977]: Version: 
'5.0.32-Debian_7etch1-log'  socket: '/var/run/mysqld/mysqld.sock'  port: 
3306  Debian etch distribution
Oct  8 17:12:59 srv-cluster-1 /etc/mysql/debian-start[5014]: Upgrading 
MySQL tables if necessary.
Oct  8 17:12:59 srv-cluster-1 ResourceManager[3468]: [5022]: debug: 
/etc/init.d/mysql  start done. RC=0
Oct  8 17:12:59 srv-cluster-1 ResourceManager[3468]: [5038]: info: 
Running /etc/init.d/apache2  start
Oct  8 17:12:59 srv-cluster-1 ResourceManager[3468]: [5039]: debug: 
Starting /etc/init.d/apache2  start
Oct  8 17:13:00 srv-cluster-1 /etc/mysql/debian-start[5044]: Checking 
for crashed MySQL tables.
Oct  8 17:13:01 srv-cluster-1 ResourceManager[3468]: [5054]: debug: 
/etc/init.d/apache2  start done. RC=0
Oct  8 17:13:01 srv-cluster-1 heartbeat: [3447]: info: all HA resource 
acquisition completed (standby).
Oct  8 17:13:01 srv-cluster-1 heartbeat: [3294]: info: Standby resource 
acquisition done [all].
Oct  8 17:13:01 srv-cluster-1 heartbeat: [5055]: debug: notify_world: 
setting SIGCHLD Handler to SIG_DFL
Oct  8 17:13:01 srv-cluster-1 harc[5055]: [5058]: info: Running 
/etc/ha.d/rc.d/status status
Oct  8 17:13:01 srv-cluster-1 mach_down[5061]: [5076]: info: 
/usr/lib/heartbeat/mach_down: nice_failback: foreign resources acquired
Oct  8 17:13:01 srv-cluster-1 mach_down[5061]: [5079]: info: mach_down 
takeover complete for node srv-cluster-2.
Oct  8 17:13:01 srv-cluster-1 heartbeat: [3294]: info: mach_down 
takeover complete.
Oct  8 17:13:01 srv-cluster-1 heartbeat: [5080]: debug: notify_world: 
setting SIGCHLD Handler to SIG_DFL
Oct  8 17:13:01 srv-cluster-1 harc[5080]: [5083]: info: Running 
/etc/ha.d/rc.d/ip-request-resp ip-request-resp
Oct  8 17:13:01 srv-cluster-1 ip-request-resp[5080]: [5086]: received 
ip-request-resp ldirectord::ldirectord.cf OK yes
Oct  8 17:13:01 srv-cluster-1 ResourceManager[5087]: [5095]: info: 
Acquiring resource group: srv-cluster-1 ldirectord::ldirectord.cf 
LVSSyncDaemonSwap::master IPaddr2::192.168.30.253/24/eth0/192.168.30.255 
IPaddr2::192.168.30.252/24/eth0/192.168.30.255 drbddisk::mysql 
Filesystem::/dev/drbd0::/mnt/mysql::ext3::defaults drbddisk::www 
Filesystem::/dev/drbd2::/mnt/www::ext3::defaults mysql apache2
Oct  8 17:13:01 srv-cluster-1 ldirectord[5107]: Invoking ldirectord 
invoked as: /etc/ha.d/resource.d/ldirectord ldirectord.cf status
Oct  8 17:13:01 srv-cluster-1 ldirectord[5107]: ldirectord stale pid 
file /var/run/ldirectord.ldirectord.cf.pid for /etc/ha.d/ldirectord.cf
Oct  8 17:13:01 srv-cluster-1 ldirectord[5107]: ldirectord is stopped 
for /etc/ha.d/ldirectord.cf
Oct  8 17:13:01 srv-cluster-1 ldirectord[5107]: Exiting with exit_status 
1: Exiting from ldirectord status
Oct  8 17:13:01 srv-cluster-1 ResourceManager[5087]: [5121]: info: 
Running /etc/ha.d/resource.d/ldirectord ldirectord.cf start
Oct  8 17:13:01 srv-cluster-1 ResourceManager[5087]: [5122]: debug: 
Starting /etc/ha.d/resource.d/ldirectord ldirectord.cf start
Oct  8 17:13:01 srv-cluster-1 ldirectord[5123]: Invoking ldirectord 
invoked as: /etc/ha.d/resource.d/ldirectord ldirectord.cf start
Oct  8 17:13:01 srv-cluster-1 ldirectord[5123]: Starting Linux Director 
v1.77.2.51 as daemon
Oct  8 17:13:01 srv-cluster-1 ResourceManager[5087]: [5127]: debug: 
/etc/ha.d/resource.d/ldirectord ldirectord.cf start done. RC=0
Oct  8 17:13:01 srv-cluster-1 ldirectord[5125]: Changed virtual server: 
192.168.30.253:80
Oct  8 17:13:01 srv-cluster-1 ldirectord[5125]: Changed virtual server: 
192.168.30.253:443
Oct  8 17:13:01 srv-cluster-1 ldirectord[5125]: Changed virtual server: 
192.168.30.253:143
Oct  8 17:13:01 srv-cluster-1 ldirectord[5125]: Changed virtual server: 
192.168.30.253:993
Oct  8 17:13:01 srv-cluster-1 ldirectord[5125]: Changed virtual server: 
192.168.30.253:110
Oct  8 17:13:01 srv-cluster-1 ldirectord[5125]: Changed virtual server: 
192.168.30.253:25
Oct  8 17:13:01 srv-cluster-1 IPaddr2[5215]: [5399]: INFO: IPaddr2 
Resource is stopped
Oct  8 17:13:01 srv-cluster-1 ResourceManager[5087]: [5424]: info: 
Running /etc/ha.d/resource.d/IPaddr2 
192.168.30.253/24/eth0/192.168.30.255 start
Oct  8 17:13:01 srv-cluster-1 ResourceManager[5087]: [5427]: debug: 
Starting /etc/ha.d/resource.d/IPaddr2 
192.168.30.253/24/eth0/192.168.30.255 start
Oct  8 17:13:02 srv-cluster-1 ldirectord[5125]: Quiescent real server: 
192.168.30.120:993 ( x 192.168.30.253:993) (Weight set to 0)
Oct  8 17:13:02 srv-cluster-1 IPaddr2[5552]: [5583]: INFO: 
/usr/lib/heartbeat/send_arp -i 200 -r 5 -p 
/var/run/heartbeat/rsctmp/send_arp/send_arp-192.168.30.253 eth0 
192.168.30.253 auto 192.168.30.253 ffffffffffff
Oct  8 17:13:02 srv-cluster-1 IPaddr2[5428]: [5587]: INFO: IPaddr2 Success
Oct  8 17:13:02 srv-cluster-1 ResourceManager[5087]: [5588]: debug: 
/etc/ha.d/resource.d/IPaddr2 192.168.30.253/24/eth0/192.168.30.255 start 
done. RC=0
Oct  8 17:13:02 srv-cluster-1 IPaddr2[5600]: [5706]: INFO: IPaddr2 
Running OK
Oct  8 17:13:02 srv-cluster-1 Filesystem[5796]: [5802]: INFO: Running 
status for /dev/drbd0 on /mnt/mysql
Oct  8 17:13:02 srv-cluster-1 Filesystem[5796]: [5806]: INFO: /mnt/mysql 
is mounted (running)
Oct  8 17:13:02 srv-cluster-1 Filesystem[5732]: [5807]: INFO: Filesystem 
Running OK
Oct  8 17:13:02 srv-cluster-1 Filesystem[5897]: [5903]: INFO: Running 
status for /dev/drbd2 on /mnt/www
Oct  8 17:13:02 srv-cluster-1 Filesystem[5897]: [5907]: INFO: /mnt/www 
is mounted (running)
Oct  8 17:13:02 srv-cluster-1 Filesystem[5833]: [5908]: INFO: Filesystem 
Running OK
Oct  8 17:13:02 srv-cluster-1 ResourceManager[5087]: [5939]: info: 
Running /etc/init.d/mysql  start
Oct  8 17:13:02 srv-cluster-1 ResourceManager[5087]: [5940]: debug: 
Starting /etc/init.d/mysql  start
Oct  8 17:13:02 srv-cluster-1 ResourceManager[5087]: [5973]: debug: 
/etc/init.d/mysql  start done. RC=0
Oct  8 17:13:02 srv-cluster-1 ResourceManager[5087]: [5989]: info: 
Running /etc/init.d/apache2  start
Oct  8 17:13:02 srv-cluster-1 ResourceManager[5087]: [5990]: debug: 
Starting /etc/init.d/apache2  start
Oct  8 17:13:02 srv-cluster-1 ResourceManager[5087]: [5999]: debug: 
/etc/init.d/apache2  start done. RC=0
Oct  8 17:13:03 srv-cluster-1 kernel: drbd0: peer( Secondary -> Unknown 
) conn( Connected -> TearDown ) pdsk( UpToDate -> DUnknown )
Oct  8 17:13:03 srv-cluster-1 kernel: drbd0: Creating new current UUID
Oct  8 17:13:03 srv-cluster-1 kernel: drbd0: Writing meta data super 
block now.
Oct  8 17:13:03 srv-cluster-1 kernel: drbd0: asender terminated
Oct  8 17:13:03 srv-cluster-1 kernel: drbd0: tl_clear()
Oct  8 17:13:03 srv-cluster-1 kernel: drbd0: Connection closed
Oct  8 17:13:03 srv-cluster-1 kernel: drbd0: conn( TearDown -> 
Unconnected )
Oct  8 17:13:03 srv-cluster-1 kernel: drbd0: receiver terminated
Oct  8 17:13:03 srv-cluster-1 kernel: drbd0: receiver (re)started
Oct  8 17:13:03 srv-cluster-1 kernel: drbd0: conn( Unconnected -> 
WFConnection )
Oct  8 17:13:07 srv-cluster-1 imapd: Connection, ip=[::ffff:192.168.30.120]
Oct  8 17:13:07 srv-cluster-1 imapd: LOGOUT, ip=[::ffff:192.168.30.120], 
rcvd=21, sent=354
Oct  8 17:13:08 srv-cluster-1 heartbeat: [3294]: WARN: node 
srv-cluster-2: is dead
Oct  8 17:13:08 srv-cluster-1 heartbeat: [3294]: info: Dead node 
srv-cluster-2 gave up resources.
Oct  8 17:13:08 srv-cluster-1 heartbeat: [3294]: info: Link 
srv-cluster-2:eth1 dead.
Oct  8 17:13:08 srv-cluster-1 ipfail: [3388]: info: Status update: Node 
srv-cluster-2 now has status dead
Oct  8 17:13:09 srv-cluster-1 ipfail: [3388]: debug: Found ping node 
192.168.30.1!
Oct  8 17:13:09 srv-cluster-1 ipfail: [3388]: info: NS: We are still alive!
Oct  8 17:13:09 srv-cluster-1 ipfail: [3388]: info: Link Status update: 
Link srv-cluster-2/eth1 now has status dead
Oct  8 17:13:09 srv-cluster-1 kernel: ocfs2_dlm: Node 1 leaves domain 
ED6A9FBBB31D41BE952126E8B1D14BA6
Oct  8 17:13:09 srv-cluster-1 kernel: ocfs2_dlm: Nodes in domain 
("ED6A9FBBB31D41BE952126E8B1D14BA6"): 0
Oct  8 17:13:10 srv-cluster-1 kernel: o2net: no longer connected to node 
srv-cluster-2 (num 1) at 172.16.0.2:7777
Oct  8 17:13:10 srv-cluster-1 ipfail: [3388]: debug: Found ping node 
192.168.30.1!
Oct  8 17:13:11 srv-cluster-1 ipfail: [3388]: info: Asking other side 
for ping node count.
Oct  8 17:13:11 srv-cluster-1 ipfail: [3388]: debug: Message [num_ping] 
sent.
Oct  8 17:13:11 srv-cluster-1 ipfail: [3388]: info: Checking remote 
count of ping nodes.
Oct  8 17:13:11 srv-cluster-1 kernel: e1000: eth1: e1000_watchdog: NIC 
Link is Down
Oct  8 17:13:16 srv-cluster-1 kernel: e1000: eth1: e1000_watchdog: NIC 
Link is Up 1000 Mbps Full Duplex, Flow Control: RX
Oct  8 17:13:18 srv-cluster-1 kernel: drbd2: PingAck did not arrive in time.
Oct  8 17:13:18 srv-cluster-1 kernel: drbd2: peer( Secondary -> Unknown 
) conn( Connected -> NetworkFailure ) pdsk( UpToDate -> DUnknown )
Oct  8 17:13:18 srv-cluster-1 kernel: drbd2: Creating new current UUID
Oct  8 17:13:18 srv-cluster-1 kernel: drbd2: asender terminated
Oct  8 17:13:18 srv-cluster-1 kernel: drbd2: short read expecting header 
on sock: r=-512
Oct  8 17:13:18 srv-cluster-1 kernel: drbd2: tl_clear()
Oct  8 17:13:18 srv-cluster-1 kernel: drbd2: Connection closed
Oct  8 17:13:18 srv-cluster-1 kernel: drbd2: Writing meta data super 
block now.
Oct  8 17:13:18 srv-cluster-1 kernel: drbd2: conn( NetworkFailure -> 
Unconnected )
Oct  8 17:13:18 srv-cluster-1 kernel: drbd2: receiver terminated
Oct  8 17:13:18 srv-cluster-1 kernel: drbd2: receiver (re)started
Oct  8 17:13:18 srv-cluster-1 kernel: drbd2: conn( Unconnected -> 
WFConnection )
Oct  8 17:13:20 srv-cluster-1 kernel: drbd1: PingAck did not arrive in time.
Oct  8 17:13:20 srv-cluster-1 kernel: drbd1: peer( Primary -> Unknown ) 
conn( Connected -> NetworkFailure ) pdsk( UpToDate -> DUnknown )
Oct  8 17:13:21 srv-cluster-1 kernel: drbd1: Creating new current UUID
Oct  8 17:13:21 srv-cluster-1 kernel: drbd1: asender terminated
Oct  8 17:13:21 srv-cluster-1 kernel: drbd1: short read expecting header 
on sock: r=-512
Oct  8 17:13:21 srv-cluster-1 kernel: drbd1: tl_clear()
Oct  8 17:13:21 srv-cluster-1 kernel: drbd1: Connection closed
Oct  8 17:13:21 srv-cluster-1 kernel: drbd1: Writing meta data super 
block now.
Oct  8 17:13:21 srv-cluster-1 kernel: drbd1: conn( NetworkFailure -> 
Unconnected )
Oct  8 17:13:21 srv-cluster-1 kernel: drbd1: receiver terminated
Oct  8 17:13:21 srv-cluster-1 kernel: drbd1: receiver (re)started
Oct  8 17:13:21 srv-cluster-1 kernel: drbd1: conn( Unconnected -> 
WFConnection )
Oct  8 17:13:22 srv-cluster-1 ldirectord[5125]: Quiescent real server: 
192.168.30.121:143 ( x 192.168.30.253:143) (Weight set to 0)
Oct  8 17:13:41 srv-cluster-1 kernel: e1000: eth2: e1000_watchdog: NIC 
Link is Down
Oct  8 17:13:43 srv-cluster-1 kernel: e1000: eth2: e1000_watchdog: NIC 
Link is Up 1000 Mbps Full Duplex, Flow Control: RX/TX
Oct  8 17:13:50 srv-cluster-1 kernel: e1000: eth1: e1000_watchdog: NIC 
Link is Down
Oct  8 17:13:53 srv-cluster-1 kernel: e1000: eth1: e1000_watchdog: NIC 
Link is Up 1000 Mbps Full Duplex, Flow Control: RX/TX
Oct  8 17:13:54 srv-cluster-1 kernel: drbd0: conn( WFConnection -> 
WFReportParams )
Oct  8 17:13:54 srv-cluster-1 kernel: drbd0: Handshake successful: DRBD 
Network Protocol version 86
Oct  8 17:13:54 srv-cluster-1 kernel: drbd0: Peer authenticated using 20 
bytes of 'sha1' HMAC
Oct  8 17:13:54 srv-cluster-1 kernel: drbd1: conn( WFConnection -> 
WFReportParams )
Oct  8 17:13:54 srv-cluster-1 kernel: drbd1: Handshake successful: DRBD 
Network Protocol version 86
Oct  8 17:13:54 srv-cluster-1 kernel: drbd1: Peer authenticated using 20 
bytes of 'sha1' HMAC
Oct  8 17:13:54 srv-cluster-1 kernel: drbd1: peer( Unknown -> Secondary 
) conn( WFReportParams -> WFBitMapS ) pdsk( DUnknown -> UpToDate )
Oct  8 17:13:54 srv-cluster-1 kernel: drbd1: Writing meta data super 
block now.
Oct  8 17:13:54 srv-cluster-1 kernel: drbd0: peer( Unknown -> Secondary 
) conn( WFReportParams -> WFBitMapS ) pdsk( DUnknown -> UpToDate )
Oct  8 17:13:54 srv-cluster-1 kernel: drbd2: conn( WFConnection -> 
WFReportParams )
Oct  8 17:13:54 srv-cluster-1 kernel: drbd2: Handshake successful: DRBD 
Network Protocol version 86
Oct  8 17:13:54 srv-cluster-1 kernel: drbd2: Peer authenticated using 20 
bytes of 'sha1' HMAC
Oct  8 17:13:54 srv-cluster-1 kernel: drbd1: conn( WFBitMapS -> 
SyncSource ) pdsk( UpToDate -> Inconsistent )
Oct  8 17:13:54 srv-cluster-1 kernel: drbd1: Began resync as SyncSource 
(will sync 655360 KB [163840 bits set]).
Oct  8 17:13:54 srv-cluster-1 kernel: drbd1: Writing meta data super 
block now.
Oct  8 17:13:55 srv-cluster-1 kernel: drbd2: peer( Unknown -> Secondary 
) conn( WFReportParams -> WFBitMapS ) pdsk( DUnknown -> UpToDate )
Oct  8 17:13:55 srv-cluster-1 kernel: drbd0: Writing meta data super 
block now.
Oct  8 17:13:55 srv-cluster-1 kernel: drbd1: peer( Secondary -> Primary )
Oct  8 17:13:55 srv-cluster-1 kernel: drbd2: Writing meta data super 
block now.
Oct  8 17:13:55 srv-cluster-1 kernel: drbd2: conn( WFBitMapS -> 
SyncSource ) pdsk( UpToDate -> Inconsistent )
Oct  8 17:13:55 srv-cluster-1 kernel: drbd2: Began resync as SyncSource 
(will sync 0 KB [0 bits set]).
Oct  8 17:13:55 srv-cluster-1 kernel: drbd2: Resync done (total 1 sec; 
paused 0 sec; 0 K/sec)
Oct  8 17:13:55 srv-cluster-1 kernel: drbd2: conn( SyncSource -> 
Connected ) pdsk( Inconsistent -> UpToDate )
Oct  8 17:13:55 srv-cluster-1 kernel: drbd2: Writing meta data super 
block now.
Oct  8 17:13:55 srv-cluster-1 kernel: drbd0: conn( WFBitMapS -> 
SyncSource ) pdsk( UpToDate -> Inconsistent )
Oct  8 17:13:55 srv-cluster-1 kernel: drbd0: Began resync as SyncSource 
(will sync 124 KB [31 bits set]).
Oct  8 17:13:55 srv-cluster-1 kernel: drbd0: Writing meta data super 
block now.
Oct  8 17:13:55 srv-cluster-1 kernel: drbd0: Resync done (total 1 sec; 
paused 0 sec; 124 K/sec)
Oct  8 17:13:55 srv-cluster-1 kernel: drbd0: conn( SyncSource -> 
Connected ) pdsk( Inconsistent -> UpToDate )
Oct  8 17:13:55 srv-cluster-1 kernel: drbd0: Writing meta data super 
block now.
Oct  8 17:14:18 srv-cluster-1 kernel: o2net: accepted connection from 
node srv-cluster-2 (num 1) at 172.16.0.2:7777
Oct  8 17:14:23 srv-cluster-1 kernel: ocfs2_dlm: Node 1 joins domain 
ED6A9FBBB31D41BE952126E8B1D14BA6
Oct  8 17:14:23 srv-cluster-1 kernel: ocfs2_dlm: Nodes in domain 
("ED6A9FBBB31D41BE952126E8B1D14BA6"): 0 1
Oct  8 17:14:25 srv-cluster-1 kernel: drbd1: Resync done (total 30 sec; 
paused 0 sec; 21844 K/sec)
Oct  8 17:14:25 srv-cluster-1 kernel: drbd1: conn( SyncSource -> 
Connected ) pdsk( Inconsistent -> UpToDate )
Oct  8 17:14:25 srv-cluster-1 kernel: drbd1: Writing meta data super 
block now.
Oct  8 17:14:38 srv-cluster-1 ipfail: [3388]: info: Link Status update: 
Link srv-cluster-2/eth1 now has status up
Oct  8 17:14:38 srv-cluster-1 ipfail: [3388]: info: Status update: Node 
srv-cluster-2 now has status init
Oct  8 17:14:38 srv-cluster-1 ipfail: [3388]: info: Status update: Node 
srv-cluster-2 now has status up
Oct  8 17:14:38 srv-cluster-1 heartbeat: [3294]: info: Heartbeat restart 
on node srv-cluster-2
Oct  8 17:14:38 srv-cluster-1 heartbeat: [3294]: info: Link 
srv-cluster-2:eth1 up.
Oct  8 17:14:38 srv-cluster-1 heartbeat: [3294]: info: Status update for 
node srv-cluster-2: status init
Oct  8 17:14:38 srv-cluster-1 heartbeat: [3294]: info: Status update for 
node srv-cluster-2: status up
Oct  8 17:14:38 srv-cluster-1 heartbeat: [3294]: debug: 
StartNextRemoteRscReq(): child count 1
Oct  8 17:14:38 srv-cluster-1 heartbeat: [3294]: debug: get_delnodelist: 
delnodelist=
Oct  8 17:14:38 srv-cluster-1 heartbeat: [6023]: debug: notify_world: 
setting SIGCHLD Handler to SIG_DFL
Oct  8 17:14:38 srv-cluster-1 harc[6023]: [6026]: info: Running 
/etc/ha.d/rc.d/status status
Oct  8 17:14:38 srv-cluster-1 heartbeat: [6029]: debug: notify_world: 
setting SIGCHLD Handler to SIG_DFL
Oct  8 17:14:38 srv-cluster-1 harc[6029]: [6032]: info: Running 
/etc/ha.d/rc.d/status status
Oct  8 17:14:39 srv-cluster-1 ipfail: [3388]: info: Status update: Node 
srv-cluster-2 now has status active
Oct  8 17:14:39 srv-cluster-1 heartbeat: [3294]: info: Status update for 
node srv-cluster-2: status active
Oct  8 17:14:39 srv-cluster-1 heartbeat: [3294]: info: all clients are 
now paused
Oct  8 17:14:39 srv-cluster-1 heartbeat: [3294]: debug: hist->ackseq =140
Oct  8 17:14:39 srv-cluster-1 heartbeat: [3294]: debug: hist->lowseq 
=139, hist->hiseq=241
Oct  8 17:14:39 srv-cluster-1 heartbeat: [3294]: debug:
Oct  8 17:14:39 srv-cluster-1 heartbeat: [6035]: debug: notify_world: 
setting SIGCHLD Handler to SIG_DFL
Oct  8 17:14:39 srv-cluster-1 harc[6035]: [6038]: info: Running 
/etc/ha.d/rc.d/status status
Oct  8 17:14:39 srv-cluster-1 ipfail: [3388]: debug: Other side is unstable.
Oct  8 17:14:39 srv-cluster-1 ipfail: [3388]: debug: Other side is now 
stable.
Oct  8 17:14:39 srv-cluster-1 ipfail: [3388]: debug: Got join message 
from another ipfail client. (srv-cluster-2)
Oct  8 17:14:39 srv-cluster-1 heartbeat: [3294]: debug: hist->ackseq =140
Oct  8 17:14:39 srv-cluster-1 heartbeat: [3294]: debug: hist->lowseq 
=139, hist->hiseq=242
Oct  8 17:14:39 srv-cluster-1 heartbeat: [3294]: debug:
Oct  8 17:14:39 srv-cluster-1 heartbeat: [3294]: info: all clients are 
now resumed
Oct  8 17:14:39 srv-cluster-1 heartbeat: [3294]: info: remote resource 
transition completed.
Oct  8 17:14:39 srv-cluster-1 ipfail: [3388]: debug: Found ping node 
192.168.30.1!
Oct  8 17:14:40 srv-cluster-1 ipfail: [3388]: info: Asking other side 
for ping node count.
Oct  8 17:14:40 srv-cluster-1 ipfail: [3388]: debug: Message [num_ping] 
sent.
Oct  8 17:14:45 srv-cluster-1 ipfail: [3388]: info: No giveup timer to 
abort.
Oct  8 17:17:01 srv-cluster-1 /USR/SBIN/CRON[6042]: (root) CMD (   cd / 
&& run-parts --report /etc/cron.hourly)
Oct  8 17:20:20 srv-cluster-1 modprobe: WARNING: Not loading blacklisted 
module ipv6


More information about the Ultramonkey-users mailing list