DEV_TX_OFFLOAD_VXLAN_TNL_TSO Offload Testing - DPDK

Question

I am working on Mellanox ConnectX-5 cards and using DPDK 20.11 with CentOS 8 (4.18.0-147.5.1.el8_1.x86_64).

I wanted to test the DEV_TX_OFFLOAD_VXLAN_TNL_TSO offload and what I want to ask is that what should the packet structure be like (I am using scapy) that I should send to the DPDK application such that this offload will come into action and perform segmentation (since it is a VXLAN_TNL_TSO).

I am modifying the dpdk-ip_fragmentation example and have added: DEV_TX_OFFLOAD_IP_TNL_TSO inside the port_conf

static struct rte_eth_conf port_conf = {
    .rxmode = {
        .max_rx_pkt_len = JUMBO_FRAME_MAX_SIZE,
        .split_hdr_size = 0,
        .offloads = (DEV_RX_OFFLOAD_CHECKSUM |
                 DEV_RX_OFFLOAD_SCATTER |
                 DEV_RX_OFFLOAD_JUMBO_FRAME),
    },
    .txmode = {
        .mq_mode = ETH_MQ_TX_NONE,
        .offloads = (DEV_TX_OFFLOAD_IPV4_CKSUM | 
                 DEV_TX_OFFLOAD_VXLAN_TNL_TSO
                 ),
    },
};

And at the ol_flags:

ol_flags |= (PKT_TX_IPV4 | PKT_TX_IP_CKSUM | PKT_TX_TUNNEL_VXLAN );

In short, to test this offload it would be great if someone can help me with 2 things:

What should the packet structure be that I should send (using scapy, such that the offload comes into action)?
Required settings to do in the DPDK example application (It is not necessary to use the ip_fragmentation example, any other example would be fine too).

I have to mark this question as missing inforamtion. Since you have not mentioned 1. DPDK version, 2. Linux Kernel, 3. Firmware (if applicable), 4. current result or error logs or logs (pkt_dump) — Vipin Varghese
I have added the information of DPDK & Kernel. There are not any error logs just that the TSO is not happening at all, even after enabling the ol_flags inside the mbuf structure. — Ameer Usman
Did you try using DEV_TX_OFFLOAD_UDP_TNL_TSO for UDP Tunnel packet offload? I am available for skype debug — Vipin Varghese
I have also enabled the DEV_TX_OFFLOAD_UDP_TNL_TSO aswell but to no avail, I cannot see the packets being segmented when I send a VxLan Packet. Sure we can do skype as well, my email is [email protected]. — Ameer Usman
based on the live debug, the request is totally different. @AmeerUsman needs TSO for inner TCP payload and not VXLAN Tunnel. I have requested Ameer to update the ticket for the same. — Vipin Varghese

Vipin Varghese Vipin Varghese · Accepted Answer · 2021-01-19T09:33:51

note: Based on the 3 hours debug session, it is been clarified the title and question shared is incorrect. Hence the question will be re-edited to reflect actual requirement as how enable DPDK port with TCP-TSO offloads for tunnelled VXLAN packets.

Answer to the first question what should be scapy settings for sending a packet to DPDK DUT for TSO and receiving segmented traffic is

Disable all TSO related offload on the SCAPY interface using ethtool -K [scapy interface] rx off tx off tso off gso off gro off lro off
Set MTU to send larger frames like 9000
Ensure to send large frames as payload but less than interface MTU.
Run tcpdump for ingress traffic with the directional flag as tcpdump -eni [scapy interface] -Q in

Answers to the second question Required settings to do in the DPDK example application is as follows

dpdk testpmd application can enable HW and SW TSO offloads based on NIC support.
next best application is tep_termination, but requires vhost interface (VM) or DPDK vhost to achieve the same.
Since the requirement is targeted for any generic application like skeleton, l2fwd, one can enable as follows

Ensure to use DPDK 20.11 LTS (to get the latest and best support for TUNNEL TSO)
In application check for tx_offload capability with dev_get_info API.
Cross-check for HW TSO for tunnelled (VXLAN) packets.
If the TSO needs to done for UDP payload, check for UDP_TSO support in HW.
configure the NIC with no-multisegment, jumbo frame, max frame len > 9000 Bytes.
Receive the packet via rx_burst, and ensure the packet is ipv4, UDP, VXLAN (tunnel) with nb_segs as 1.
modify the mbuf to point to l2_len, l3_len, l4_len.
mark the packets ol_flags as PKT_TX_IPV4 | PKT_TX_TUNNEL_VXLAN | PKT_TX_TUNNEL_IP. For UDP inner payload PKT_TX_TUNNEL_UDP.
then set the segment size as DPDK MTU (1500 as default) - l3_len - l4_len

This will enable the PMD which support HW TSO offload to update appropriate fields in descriptors for the given payload to transmitted as multiple packets. For our test case scapy send packet of 9000 bytes will be converted into 7 * 1500 byte packets. this can be observed as part tcpdump.

Note:

the reference code is present in tep_termination and test_pmd.
If there is no HW offload, SW library of rte gso is available.
For HW offload all PMD as of today require the MBUF is a continuous single non-external buffer. So make sure to create mbufpool or mempool with sufficient size for receiving large packets.

DEV_TX_OFFLOAD_VXLAN_TNL_TSO Offload Testing - DPDK

1 Answers