Restarting network with keepalived on RedHat / CentOS

June 8th, 2013 by bostjan


This was supposed to be a normal setup: highly-available gateways/firewall with CentOS and keepalived. element iphone 8 case ghostek iphone 8 plus case Been there, done that, just not with CentOS. shock case iphone 7 Anyway, I anticipated no problems whatsoever. iphone 7 case dalmatian ravenclaw iphone 6 case iphone 6 silicone case purple

The Twist

The installation went smooth, firewall configuration was deployed from git from previous server, and so was keepalived.conf. teen iphone 8 plus case doctor who iphone 7 plus case Startup normal, things worked as expected. mustard iphone 7 case fall out boy phone case iphone 7 iphone 7 case checkerboard gudetama iphone 8 plus case iphone 7 phone case dalmation Then there was this thing with removing old statically-configured IP addresses from the system. iphone 6 case gold and black banksy phone case iphone 8 iphone x slim case I edited appropriate files in /etc/sysconfig/network-scripts/, executed ‘service network restart’ and was cut off from the machine. air jordan iphone 6 case iphone 7 phone cases greys anatomy iphone 7 phone cases for girls 3 in 1 phone case for iphone 8 WTH? This almost never happened before on RH/CentOS. iphone 7 armour case spigen phone cases iphone 6 with screen protector iphone 7 phone case led case Ok, it did happen when I messed up the IP digits, or when I tried to restart only one interface (dunno exactly about this one, but machine was 200km away and I really did not feel like roadtrip so I just resorted to restarting whole network stack which seemed to work ATM). element case iphone 7 quirky phone case iphone 6 iphone 7 case with purse Fortunately I had a backup network interface still reachable and thorough that I was able to restore connections and debug the situation. holographic marble iphone 6 case iphone 6 white marble case

The Reason

It turned out that keepalived was lagging a little bit behind actual system state and it tried to follow the network interface status changes, but it actually messed up the networking configuration. llama iphone 7 plus case mickey mouse case iphone 7 plus iphone 7 phone case purple iphone 7 phone cases blue silicone Here is what should happen:

  1. ‘service network restart’ executed
  2. interfaces go down
  3. interfaces go up
  4. keepalived does its magic to assign correct HA IP addresses

Here is what actually happened:

  1. ‘service network restart’ executed
  2. interfaces go down
  3. interfaces go up
  4. as soon as interfaces get up, keepalived readds HA IP addresses
  5. primary IP address then gets assigned to the interface as secondary address
  6. keepalived receives higher-priority VRRP advertisement, and removes IP address from interface
  7. unfortunately this also removes should-be-primary IP address and leaves server with an IP-address-less interface and thus unreachable server

The Resolution

The resolution was simple: do not restart network when keepalived is running. girls iphone 7 case rose gold iphone 7 plus case with ring 3 in one phone case iphone 6 iphone 6 phone case space iphone 7 plus cases shock proof best iphone 6 plus cases To achieve that we needed to modify the /etc/init.d/network script which now notifies the admin that keepalived is running and refuses to continue in such situation. iphone 7 plus phone case gold iphone 7 photo phone cases husky iphone 7 case gear4 iphone case 6 plus Here is the diff of the changes we did: network-keepalived.diff I hope no one else gets biten by this peculiarity.

Tags: , , , , , , ,

Leave a Reply