High-availability relay

Introduction to the high-availability relay

  • A high-availability relay setup enables a backup in case of failure.
  • We use the router software keepalived to create a high availability infrastructure.
  • It is best to have both relays fully configured before starting this process.

To configure a relay, see the In-house Relay installation using the software package article. There are several steps required to configuring an HA relay.

Configuring the high-availability relay

The following steps describe the process to configure a high-availability relay:

Define the primary and secondary relays

First, you should decide which of the relays will be the primary relay and which will be the secondary relay.

Install the keepalived package on both relays

Perform this procedure on both the primary and secondary relays.

  • Add the keepalived paths to your sources.list, then install keepalived on the relay.

    $ sudo tee -a /etc/apt/sources.list <<EOF
    deb http://ppa.launchpad.net/keepalived/stable/ubuntu precise main
    deb-src http://ppa.launchpad.net/keepalived/stable/ubuntu precise main
    EOF
    $ sudo apt-get update 
    $ sudo apt-get install keepalived
  • However, if keepalived is already installed on the relay, you should just create a backup of the current .conf file.

    $ sudo cp /etc/keepalived/keepalived.conf /etc/keepalived/keepalived.conf.backup

Configuring the primary relay

  • Create a new file /etc/keepalived/keepalived.conf or download it from here and edit it:

    vrrp_script chk_relay {
        script "/etc/keepalived/chk_relay.sh"
        interval 1
    }
    vrrp_instance VI_1 {
        state MASTER
        nopreempt
        interface eth0
        virtual_router_id 1
        unicast_src_ip 10.0.2.15
        unicast_peer {
            10.0.2.16
        }
        priority 100
        advert_int 1
        authentication {
            auth_type PASS
            auth_pass IQ$Sg&39t(9!KOM
        }
        track_script {
            chk_relay
        }
        virtual_ipaddress {
            10.0.2.252 brd 10.0.2.255 dev eth0
        }
    }
  • Modify the following fields in the .conf file to reflect your current configuration:
    • Interface - The networking ID in your relay (for example, eth0)
    • unicast_src_ip - The static IP address set for this relay (for example, 10.0.2.15)
    • unicast_peer - The static IP address of the backup relay (for example, 10.0.2.16)
    • virtual_ipaddress - The virtual IP address that devices will send to and the IP address Devo will see when data is sent.
      • [VIRTUAL_IP]/24 brd [IP_BROADCAST] dev [INTERFACE] (for example, 10.0.2.252/24 brd 10.0.2.255 dev eth0)  
    •  auth_pass - The password must be the same for both relays. We recommend changing the password.

Configuring the secondary relay

  • Create a new file /etc/keepalived/keepalived.conf or download it from here and edit it:

    vrrp_script chk_relay {
        script "/etc/keepalived/chk_relay.sh"
        interval 1
        weight 2
    }
     
    vrrp_instance VI_2 {
        state BACKUP
        interface eth0
        virtual_router_id 1
        unicast_src_ip 10.0.2.16
        unicast_peer {
            10.0.2.15
        }
     
        priority 50
        advert_int 4
        authentication {
            auth_type PASS
            auth_pass IQ$Sg&39t(9!KOM
        }
        track_script {
            chk_relay
        }
        virtual_ipaddress {
            10.0.2.252 brd 10.0.2.255 dev eth0
        }
    } 
  • Modify the following fields in the .conf file to reflect your current configuration:
    • Interface - The networking ID in your relay (for example, eth0)
    • unicast_src_ip - The static IP address set for this relay (for example, 10.0.2.15)
    • unicast_peer - The static IP address of the backup relay (for example, 10.0.2.16)
    • virtual_ipaddress - The virtual IP address that devices will send to and the IP address Devo will see when data is sent.
      • [VIRTUAL_IP]/24 brd [IP_BROADCAST] dev [INTERFACE] (for example, 10.0.2.252/24 brd 10.0.2.255 dev eth0)  
    •  auth_pass - The password must be the same for both relays. We recommend changing the password.

Configure chk_relay.sh on both relays

  • Create a file under /etc/keepalived/chk_relay.sh or download it from here.

    #!/bin/bash
    
    relay_pid=`cat /var/run/scoja.pid`
    
    lsof -i|grep 13000|grep LISTEN
    
    if [ "$?" -eq "1" ] || [ -z $relay_pid ]; then
            logger  "relay stopped"
            number_of_proccess=`ps -aux|grep -v grep|grep -c java`
    
            if [ "$number_of_proccess" -lt "1"  ]; then
                    /etc/init.d/logtrust-relay start
                    sleep 4
                    lsof -i|grep 13000|grep LISTEN
                    relay_pid=`cat /var/run/scoja.pid`
                    if [ "$?" -eq "1" ]   &&  [ -z $relay_pid ]; then
                            logger  "relay stopped"
                    else
                            logger "relay is back up "
                    fi
            fi
    fi
  • Make sure to monitor the port your sources are sending to.
  • Note that by default we monitor port 13000 since we are assuming that sources are sending data properly tagged and no relay rules are necessary.
  • If that is not the case, then adjust the document to match the port your sources are sending to. 

Grant permissions to run the script

$ sudo chmod +x /etc/keepalived/chk_relay.sh 
  • This script will be watching if port 13000 is up on the relay.

Restart keepalived with new configuration

  • Now that both relays are properly configured, restart the keepalived process in each relay.

    $ sudo /etc/init.d/keepalived restart
  • Check if one of the relays has obtained the virtual IP you have previously assigned in the interface (that is, eth0 10.0.2.252).

Testing Keepalived

After restarting the keepalived process, you should perform a few tests to make sure it is functioning correctly. 

Confirm that the virtual IP is displayed in the addresses for the primary relay

$ip a 

The primary relay IP and the virtual IP should be part of the command response.


Now you want to do the same with the secondary relay. To do this, you need to restart keepalived in the primary relay, thus making the secondary relay the "master" relay.

$ sudo /etc/init.d/keepalived restart && tail -n 100 -f /var/log/syslog

Or

$ sudo service keepalived restart && tail -n 100 -f /var/log/syslog

The command response will confirm that the service is restarting. This temporarily turns the primary into the backup relay, making the secondary relay the "master" relay. Now you can carry out step 2.

Ensure that the virtual IP address is active in the secondary relay

$ip a 

The secondary relay IP and the virtual IP should be part of the command response. 

Now, restart the keepalived process in the secondary relay, to make the primary relay the "master" relay again. 

$ sudo /etc/init.d/keepalived restart && tail -n 100 -f /var/log/syslog

Or

$ sudo service keepalived restart && tail -n 100 -f /var/log/syslog 

You have now confirmed that the keepalived process is working correctly in both relays.

Send test data to the virtual IP

We can use netcat to send 100 events to the virtual IP to confirm that the relays are processing events correctly and forwarding them to Devo. 

(for i in `seq 1 100`; do echo "<14>Jan 10 10:00:00 xxx test.keep.free: test event $i"; done|nc localhost 13000); sleep 1 

Change localhost in the command above to match your virtual IP. To confirm that the events were processed correctly, open Devo, go to Data Search, and locate the test.keep.free table. This table should contain 100 events. 

Using Keepalived in Ubuntu 16.04

Ubuntu 16.04 requires some changes to the standard installation process. This is because Ubuntu 16.04 does not come with Python PIP, and one of the scripts needs it to change the IP.   

Note that Ubuntu 16.10 is not affected, so no adjustments are required.
  • First, install Python PIP using the following command:

    $ sudo apt-get install python-pip && export LC_ALL=C && pip install requests
  • Now, go to the keepalived.service file, then open it for editing.

    $ sudo nano /etc/systemd/system/keepalived.service
  • Replace its contents with the information below, then save the file: 

    #
    # keepalived control files for systemd
    #
    # Incorporates fixes from RedHat bug #769726.
    [Unit]
    Description=LVS and VRRP High Availability monitor
    After=network.target
    ConditionFileNotEmpty=/etc/keepalived/keepalived.conf
    [Service]
    Type=simple
    # Ubuntu/Debian convention:
    EnvironmentFile=-/etc/default/keepalived
    ExecStart=/usr/sbin/keepalived --dont-fork
    ExecReload=/bin/kill -s HUP $MAINPID
    # keepalived needs to be in charge of killing its own children.
    KillMode=process
    [Install]
    WantedBy=multi-user.target
  • Start the keepalived service using the following command.

    $ sudo systemctl start keepalived.service
  • To ensure that the keepalived service has started, you can view the most recent events in the syslog.

    $ sudo tail –n 100 –f /var/log/syslog


Have we answered your question?

If not, please contact our technical support team via email by clicking the button below.

CONTACT US