rx_queue_index returns always value 1 for ipv6-Srv6 packets on XDP-eBPF

Question

I am trying to keep track of Ipv6-Srv6 packets using simple XDP-eBPF program. I get the NIC queue as always 1.

Following is environment details

OS: guest OS, Ubuntu 19.10, 5.3.0-64-generic
NIC: x710: 10Gbps, driver=i40e, driverversion=2.8.20-k, firmware=7.20
XDP-eBPF: my custom sample program

#include <bpf.h>
#include <bpf/bpf_helpers.h>

#include <linux/if_ether.h>
#include <linux/ipv6.h>

#include <stdint.h>

#ifndef lock_xadd
#define lock_xadd(ptr, val)     ((void) __sync_fetch_and_add(ptr, val))
#endif

struct bpf_map_def SEC("maps") queue_rx_pkts = {
        .type        = BPF_MAP_TYPE_ARRAY,
        .key_size    = sizeof(__u32),
        .value_size  = sizeof(__u64),
        .max_entries = 256,
};

SEC("prog")
int xdp_select(struct xdp_md *ctx)
{
        //__u32 input_port = ctx->ingress_ifindex;
        __u32 input_queue = ctx->rx_queue_index;

        void *data_end = (void *)(long)ctx->data_end;
        void *data = (void *)(long)ctx->data;
        struct ethhdr *eth_ptr = (struct ethhdr *)data;
        struct ipv6hdr *ipv6_ptr = (struct ipv6hdr *)(data + sizeof(struct ethhdr));
        unsigned int *pkt_count = bpf_map_lookup_elem(&queue_rx_pkts, &input_queue);

        if (!pkt_count)
                return XDP_ABORTED;

        if ((data + sizeof(*eth_ptr)) > data_end)
                return XDP_DROP;


        if (eth_ptr->h_proto == 0xdd86) {
                if ((void *)(ipv6_ptr + 1) > data_end)
                        return XDP_DROP;
                *pkt_count += 1;
                return XDP_PASS;
        }

        return XDP_DROP;
}

packet generator: Host NIC X710 run with packEth cli utility with packETHcli -i ens261f2 -m 2- d 0 -n 0 -f /tmp/test.pcap
Expected result: Ingress packets would be spread on multiple RX queues.
Observed result: RX queue 1 is only getting packets.

I verified the results with ethtool -S [interface name] and bpftool map dump id 12. Both cases RX-Queue-1 is only getting updated.

[EDIT-1] updating RSS hash status as per the recommendation from comment

$ ethtool --show-rxfh-indir ens10                                                                                                                                                                                                      [4/1785]
RX flow hash indirection table for ens10 with 8 RX ring(s):
    0:      0     1     2     3     4     5     6     7
    8:      0     1     2     3     4     5     6     7
<snipped>
  504:      0     1     2     3     4     5     6     7
RSS hash key:
9b:d2:e1:c0:f7:aa:1f:69:49:7c:55:76:48:7e:2c:fc:d5:91:47:39:df:50:52:8f:88:07:c8:63:62:27:93:17:35:75:dc:f6:85:e8:ff:c2:72:1d:de:92:d4:2b:1e:d9:f2:53:df:38
RSS hash function:
    toeplitz: on
    xor: off
    crc32: off

Thank you @Progman for the edit, but do you feel the C code is causing the issue? — Vipin Varghese
It looks like the packet generator in the example of yours transmits the very same packet all the time. Is that the case? If yes, why do you expect the packets to spread across multiple queues? Say, the hash function enabled on the device is "RSS-IPv6". Then packet fields contributing to the packet hash are the source and destination addresses. They're constant across the packets, hence the same hash value, hence the packets hitting the same queue all the time. Does this explanation make sense? — stackinside
@stackinside thank you it does make sense, can you please post it as answer. I will check the same by changing IP address and update the question as required — Vipin Varghese
Well, I believe one more point to check would be the actual set of hash functions enabled on the NIC. To make sure that RSS-IPv6 is enabled. — stackinside

stackinside stackinside · Accepted Answer · 2021-02-06T11:08:03

It looks like the packet generator in the example of yours transmits the very same packet all the time. Is that the case? If yes, why do you expect the packets to spread across multiple queues? Say, the hash function enabled on the device is "RSS-IPv6". Then packet fields contributing to the packet hash are the source and destination addresses. They're constant across the packets, hence the same hash value, hence the packets hitting the same queue all the time.

Based on OP's further observations, enabling IP address randomisation in PackETH (putting a tick at the option Set random source IPv6 address with mask and, in example, using the mask value of 64) indeed results in generated packets hitting different Rx queues on the receiver side. Per-queue packet statistics obtained by means of ethtool -S confirm this observation.

rx_queue_index returns always value 1 for ipv6-Srv6 packets on XDP-eBPF

1 Answers