0
votes

ISSUE description

I have a OpenStack system with HA management network (VIP) via ovs (Open vSwitch) port, it's found in this system, with high load (concurrently volume-from-glance-image creation), the VIP port (an ovs port) will be missing.

Analysis

For now, with default log level from log file, the only thing observed is as below the Unreasonably long 62741ms poll interval.

2017-12-29T16:40:38.611Z|00001|timeval(revalidator70)|WARN|Unreasonably long 62741ms poll interval (0ms user, 0ms system)

Idea for now

I will turn debug log on for file and try reproducing the issue:

sudo ovs-appctl vlog/set file:dbg

Question

  • What else should I do during/after of the issue reproduction please?
  • Is this issue typical? Caused by what if yes?

I googled OpenvSwitch trouble shoot or other related key words while information was all on data flow/table level instead of this ovs-vswitchd level ( am I right? )

Many thanks! BR//Wey

1
What Linux kernel version and what OVS version? By HA, are you using LACP or? And at the OVS level or in the nic? Are you using STP?gdahlm
@gdahlm it's a virtual openstack controller Ubuntu 14.04 guest running via kvm, I dont have those version info yet. Regarding HA, it's actually a mirantis openstack : ocf::fuel:ns_IPaddr2 pacemaker resource. Regirding STP I dont know for now yet, will update later. Thanks!Wey Gu
OVS that shipped on 14.04 doesn't handle load very well, as mirantis isolates the systems from the normal package management flow I don't know what version, but see if it is at least ovs 2.5.1. But if it is using the ubuntu cloud archive, the updated version and adding the Data Plane Development Kit (DPDK) will help out a lot. There are some numa pinning and other options that may help you a bit, but a newer ovs and dpdk really help with the ml2 driver model.gdahlm
@gdahlm it's using 2.6.1 with DPDK (pmd on dedicated cpu on both numa nodes) and numa awared cpu pinning. Thanks! I will try reproducing with debug log level to see what was found.Wey Gu

1 Answers

0
votes

This issue was not reproduced and thus I forgot about it, until recently, 2 years afterward, I had a chance to get in touch with this issue in a different environment, and this time I have more ideas on its root cause.

It could be caused by the shift that comes in the bonding, for some reason, the traffic pattern fits the situation of triggering shifts again and again(the condition is quite strong I would say but there is a chance to be hit anyway, right?).

The condition of the shift was quoted as below and please refer to the full doc here: https://docs.openvswitch.org/en/latest/topics/bonding/

Bond Packet Output When a packet is sent out a bond port, the bond member actually used is selected based on the packet’s source MAC and VLAN tag (see bond_choose_output_member()). In particular, the source MAC and VLAN tag are hashed into one of 256 values, and that value is looked up in a hash table (the “bond hash”) kept in the bond_hash member of struct port. The hash table entry identifies a bond member. If no bond member has yet been chosen for that hash table entry, vswitchd chooses one arbitrarily.

Every 10 seconds, vswitchd rebalances the bond members (see bond_rebalance()). To rebalance, vswitchd examines the statistics for the number of bytes transmitted by each member over approximately the past minute, with data sent more recently weighted more heavily than data sent less recently. It considers each of the members in order from most-loaded to least-loaded. If highly loaded member H is significantly more heavily loaded than the least-loaded member L, and member H carries at least two hashes, then vswitchd shifts one of H’s hashes to L. However, vswitchd will only shift a hash from H to L if it will decrease the ratio of the load between H and L by at least 0.1.

Currently, “significantly more loaded” means that H must carry at least 1 Mbps more traffic, and that traffic must be at least 3% greater than L’s.