KB 2725: The Olfeo server sends Keepalived (VRRP) failover alerts

The Olfeo server issues keepalived failover alerts.

Symptom

The Olfeo server issues keepalived failover alerts:

Background

When creating a cluster in Settings > High Availability > Clusters, you must declare a virtual IP address for each member. When a node in the Olfeo cluster is unavailable, its virtual IP address is then switched to its backup node. The server then sends an email to alert the administrator.

The service that manages high availability is Keepalived (uses the VRRP protocol). The virtual IP address of a cluster member is switched if:

• the server proxy is no longer responding
• The server's Keepalived service is no longer responding.

We invite you to use the /olfeo-status.sh script to display the status of services:

Steps

Step 1: Proxy malfunction

You need to analyze the /opt/olfeo5/data/proxy/log/cache.log file to find out if the proxy has been shut down/restarted. FATAL messages should be monitored as they indicate that the proxy has stopped. When the cluster is configured, the server issues a keepalived failover alert in Settings > Monitoring whenthe proxy stops or restarts.

See also: Squid documentation on logs.

Step 2: Malfunction related to Keepalived

If the problem is not caused by the proxy, it means that the Keepalived process itself is experiencing an issue. The source of the malfunction must be identified using a frame capture.

tcpdump -ni any -s0 -C 50 -W 5 arp or vrrp -w vrrp_vrrp.pcap

The above command must be executed on both the master and slave. It is a circular tcpdump that generates a maximum of 5 files of 50 MB each, capturing VRRP and ARP frames. The command must be run before the failover.

In the event of a switchover, the following messages appear in /var/log/syslog:

Example of master server frames:

The image above indicates that for 18 seconds, the master server was no longer receiving VRRP advertisements from the slave.

The master server then decides to send a gratuitous ARP request to announce that it holds the slave's virtual IP address (10.0.255.14), and Olfeo triggers the sending of an email alert (e.g., the one at 2:00:20 p.m. (UTC)) if the Send email alerts box is checked in Settings > Monitoring:

Wireshark filter = arp.isgratuitous

You must also analyze the result of the slave's tcpdump command to see if it was sending VRRP announcements at regular intervals (check if, for example, the 18 seconds are missing). If this is the case, the switchover could be caused by a network problem (server unreachable).

Please do not hesitate to open a support ticket if Keepalived is not sending VRRP announcements.

Wireshark documentation: https://wiki.wireshark.org/Gratuitous_ARP.

Resolution

Once the situation has been resolved, the alerts should disappear.